The dme_peer get/set attribute commands are prone to errors, therefore
we add three retries for the UIC command sending.  Error code returned
from ufshcd_send_uic_cmd() is checked, and unless it was successful or
the retries have finished, another command will be sent.
Reviewed-by: NGilad Broner <gbroner@codeaurora.org>
Reviewed-by: NDolev Raviv <draviv@codeaurora.org>
Signed-off-by: NLee Susman <lsusman@codeaurora.org>
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

If device raises the exception event in the response to the commands
sent during the runtime/system PM callbacks, exception event handler
might run in parallel with PM callbacks and may see unclocked register
accesses. This change fixes this issue by not scheduling the exception
event handler while PM callbacks are running.
Reviewed-by: NGilad Broner <gbroner@codeaurora.org>
Reviewed-by: NDolev Raviv <draviv@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

fDeviceInit query response time for some devices is too long that
default query request timeout of 100ms may not be enough. Experiments
show that fDeviceInit response sometimes takes 500ms so to be on safer
side this change sets the timeout to 600ms. Without this change, we
might unnecessarily have to retry fDeviceInit query requests multiple
times and each query request timeout prints one error message.
Reviewed-by: NGilad Broner <gbroner@codeaurora.org>
Reviewed-by: NDolev Raviv <draviv@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

When sending a query to the device returns with a timeout error, we
clear the corresponding bit in the DOORBELL register but we don't clear
the outstanding_request field as we should.  This patch fixes this bug.
Reviewed-by: NSubhash Jadavani <subhashj@codeaurora.org>
Reviewed-by: NGilad Broner <gbroner@codeaurora.org>
Reviewed-by: NDolev Raviv <draviv@codeaurora.org>
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

A race condition appear to exist between request completion when
scsi_done() is called to end the request and set the tag back to -1 (at
blk_queue_end_tag() scsi_end_request), and scsi layer error handling
which aborts the command and reuses it to request sense data. Sending
the request sense is done with tag which was set to -1 and so it is
invalid.  Assert command tag passed from scsi layer is valid.
Reviewed-by: NSubhash Jadavani <subhashj@codeaurora.org>
Reviewed-by: NDolev Raviv <draviv@codeaurora.org>
Signed-off-by: NGilad Broner <gbroner@codeaurora.org>
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Some of the data structures (like response UPIU) and/or its elements
(unused fields) should be cleared before sending out the respective
command to UFS device.

This change clears the UPIU response data structure for query commands
and NOP command before sending out the command. We also initialize the
PRDT table length to zero which should take care of commands which
doesn't have any data associated with it. We are also clearing the
unused fields in request UPIU for NOP command.
Reviewed-by: NGilad Broner <gbroner@codeaurora.org>
Reviewed-by: NDolev Raviv <draviv@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Tested-by: NTomas Winkler <tomas.winkler@intel.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Clear the UFS data structures before sending new request.

The SCSI command is sent to the device within the UFS UPIU request.
As part of the transfer UPIU preparation, the SCSI command is copied
to the UPIU structure according to the SCSI command size.
As different SCSI commands differ in size from each other, we need
to clear the whole SCSI command field to prevent sending uninitialized
data to the device.

The UPIU response doesn't always include the sense data and can differ
in size.
Hence, the UPIU response should also be cleared before the transfer.
Reviewed-by: NGilad Broner <gbroner@codeaurora.org>
Reviewed-by: NDolev Raviv <draviv@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMaya Erez <merez@codeaurora.org>
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

This patch changes the !blk-mq path to the same defaults as the blk-mq
I/O path by always enabling block tagging, and always using host wide
tags.  We've had blk-mq available for a few releases so bugs with
this mode should have been ironed out, and this ensures we get better
coverage of over tagging setup over different configs.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NJens Axboe <axboe@kernel.dk>
Reviewed-by: NHannes Reinecke <hare@suse.de>
Signed-off-by: NJames Bottomley <JBottomley@Odin.com>

New revisions of UFS host controller supports the new UniPro
hardware controller (referred as QUniPro). This patch adds
the support to enable this new UniPro controller hardware.

This change also adds power optimization for bus scaling feature,
as well as support for HS-G3 power mode.
Reviewed-by: NSubhash Jadavani <subhashj@codeaurora.org>
Reviewed-by: NGilad Broner <gbroner@codeaurora.org>
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Reviewed-by: NHannes Reinecke <hare@suse.de>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Adds support for configuring and reading the test bus and debug
registers. This change also adds another vops in order to print the
debug registers.
Reviewed-by: NSubhash Jadavani <subhashj@codeaurora.org>
Reviewed-by: NGilad Broner <gbroner@codeaurora.org>
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Reviewed-by: NHannes Reinecke <hare@suse.de>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

This change turns the UFS variant (SCSI_UFS_QCOM) into a UFS
a platform device.
In order to do so a few additional changes are required:
1. The ufshcd-pltfrm is no longer serves as a platform device.
   Now it only serves as a group of platform APIs such as PM APIs
   (runtime suspend/resume, system suspend/resume etc), parsers of
   clocks, regulators and pm_levels from DT.
2. What used to be the old platform "probe" is now "only"
   a pltfrm_init() routine, that does exactly the same, but only
   being called by the new probe function of the UFS variant.
Reviewed-by: NRob Herring <robherring2@gmail.com>
Reviewed-by: NGilad Broner <gbroner@codeaurora.org>
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Tested-by: NAlim Akhtar <alim.akhtar@samsung.com>
Reviewed-by: NHannes Reinecke <hare@suse.de>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

In order to simplify the code a set of wrapper functions is created
to test and call each of the variant operations.
Reviewed-by: NAkinobu Mita <akinobu.mita@gmail.com>
Reviewed-by: NSubhash Jadavani <subhashj@codeaurora.org>
Reviewed-by: NGilad Broner <gbroner@codeaurora.org>
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Tested-by: NAlim Akhtar <alim.akhtar@samsung.com>
Reviewed-by: NHannes Reinecke <hare@suse.de>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Some host controller hardware controllers may not advertise correct
version in UFS HCI VER register. To workaround this, add new quirk
and call the host controller hardware vendor specific callback to
get the correct UFS HCI version register value.
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Reviewed-by: NAkinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: NJames Bottomley <JBottomley@Odin.com>

Some UFS host controllers may only allow accessing the peer DME attribute
in AUTO mode (FAST AUTO or SLOW AUTO) hence we had added a quirk for
switching to AUTO power mode before accessing the peer DME attribute.
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Reviewed-by: NAkinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: NJames Bottomley <JBottomley@Odin.com>

The attribute PA_RXHSUNTERMCAP specifies whether or not the
inbound Link supports unterminated line in HS mode. enabling this
attribute to 1 fixes moving to HS gear.
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Reviewed-by: NAkinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: NJames Bottomley <JBottomley@Odin.com>

LCC (Line Control Command) are being used for communication between
UFS host and UFS device.
New commercial UFS devices don't have the issues with LCC processing
but UFS host controller might still have the issue with LCC processing,
hence, added a routine to disable TX LCC on the device.
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Reviewed-by: NAkinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: NJames Bottomley <JBottomley@Odin.com>

UFS HCI (Host Controller Interface) allows the transfer requests
interrupts to be aggregated to generate the single interrupt but
this can impact the performance. Hence introduce the capability which
gives choice to use the interrupt aggregation capability or not.
By default interrupt aggregation capability is kept disabled.

This change also introduces a quirk for broken interrupt aggregation
feature, as in some UFS controllers, this feature may not work.
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Reviewed-by: NAkinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: NJames Bottomley <JBottomley@Odin.com>

Some implementation of UFS host controller HW might have some non-standard
behaviours (quirks) when compared to behaviour specified by UFSHCI
specification. This patch add support to allow specifying all such quirks
to standard UFS host controller driver so standard driver takes them into
account.

In this change a UFSHCD_QUIRK_DELAY_BEFORE_DME_CMDS is introduced,
where a minimum delay of 1ms is required before DME commands for
stability purposes.
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Reviewed-by: NGilad Broner <gbroner@codeaurora.org>
Signed-off-by: NJames Bottomley <JBottomley@Odin.com>

The function regulator_set_optimum_mode() is changing name to
regulator_set_load(), so update the code accordingly. Also cleaned up
ufshcd_config_vreg_load() while touching the code.
Signed-off-by: NBjorn Andersson <bjorn.andersson@sonymobile.com>
Signed-off-by: NMark Brown <broonie@kernel.org>

Convert sense buffer logging to use the per-cpu buffer to avoid line
breakup.
Tested-by: NRobert Elliott <elliott@hp.com>
Reviewed-by: NRobert Elliott <elliott@hp.com>
Signed-off-by: NHannes Reinecke <hare@suse.de>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

If no voltage supply regulators are defined for the UFS devices (assumed
they are always-on), ufshcd_config_vreg_load() can be called on
suspend/resume paths with vreg == NULL as hba->vreg_info.vcc* equal to
NULL, and it causes NULL pointer dereference.

This fixes it by making ufshcd_config_vreg_{h,l}pm noop when no regulators
are defined.
Signed-off-by: NAkinobu Mita <mita@fixstars.com>
Reviewed-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

When dynamic clk gating feature is enabled, delayed workqueue machanism
is used in order to detect certain period of inactivity.  But there is no
guarantee that scheduled gating work is completed before module unloading.
So it can cause kernel crash by accessing memory after it was freed.

Fix it by cancelling clk gating and ungating works and ensure that its
execution is finished before module unloading.
Signed-off-by: NAkinobu Mita <mita@fixstars.com>
Reviewed-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

Drop the now unused reason argument from the ->change_queue_depth method.
Also add a return value to scsi_adjust_queue_depth, and rename it to
scsi_change_queue_depth now that it can be used as the default
->change_queue_depth implementation.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NMike Christie <michaelc@cs.wisc.edu>
Reviewed-by: NHannes Reinecke <hare@suse.de>

All drivers use the implementation for ramping the queue up and down, so
instead of overloading the change_queue_depth method call the
implementation diretly if the driver opts into it by setting the
track_queue_depth flag in the host template.

Note that a few drivers validated the new queue depth in their
change_queue_depth method, but as we never go over the queue depth
set during slave_configure or the sysfs file this isn't nessecary
and can safely be removed.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NMike Christie <michaelc@cs.wisc.edu>
Reviewed-by: NHannes Reinecke <hare@suse.de>
Reviewed-by: NVenkatesh Srinivas <venkateshs@google.com>

This patch fixes newly introduced static checker warning in
__ufshcd_setup_clocks, introduced by UFS power management series.

Warning:
drivers/scsi/ufs/ufshcd.c:4474 __ufshcd_setup_clocks()
warn: we tested 'ret' before and it was 'false'

To fix it we remove the (!ret) from the condition.
Signed-off-by: NDolev Raviv <draviv@codeaurora.org>
Reviewed-by: NMaya Erez <merez@codeaurora.org>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

This patch fixes newly introduced sparse warning in
ufshcd_system_suspend, introduced by UFS power management series.

Sparse warning:
drivers/scsi/ufs/ufshcd.c:5118 ufshcd_system_suspend()
error: we previously assumed 'hba' could be null (see line 5089)

To fix it, we return 0 in case HBA is not initialized or is
not powered.
Signed-off-by: NDolev Raviv <draviv@codeaurora.org>
Reviewed-by: NMaya Erez <merez@codeaurora.org>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

After link start-up power mode will always be PWM G1. This is not
reflected in the pwr_info struct which will keep the previous values.
Since ufshcd_change_power_mode() tries to avoid unnecessary power mode
change if the requested power mode and current power mode are same,
power mode change won't execute again after driver initialization.

This patch solves the problem by setting pwr_info to PWM G1 after link
start-up.
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Signed-off-by: NDolev Raviv <draviv@codeaurora.org>
Reviewed-by: NMaya Erez <merez@codeaurora.org>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

UFS driver adds three well known LUs in the initialization, but those
reference counts are not decremented, so it makes ufshcd module
impossible to unload.

This fixes it by putting scsi_device_put() in the initalization, and in
order to protect concurrent access to hba->sdev_ufs_device (UFS Device
W-LU) from manual delete, increment the reference count while requesting
device power mode setting.

The rest of W-LUs (hba->sdev_boot and hba->sdev_rpmb) are not directly
used from driver, so these references in struct ufs_hba are removed.
Signed-off-by: NAkinobu Mita <mita@fixstars.com>
Reviewed-by: NMaya Erez <merez@codeaurora.org>
Reviewed-by: NDolev Raviv <draviv@codeaurora.org>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

ufs never looks at the tag type, so there is no need to set it either.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NMike Christie <michaelc@cs.wisc.edu>
Reviewed-by: NHannes Reinecke <hare@suse.de>

Now that we also get proper values in cmd->request->tag for untagged
commands, there is no need to force tagged_supported to on in drivers
that need host-wide tags.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NMike Christie <michaelc@cs.wisc.edu>
Reviewed-by: NHannes Reinecke <hare@suse.de>

Remove the tagged argument from scsi_adjust_queue_depth, and just let it
handle the queue depth.  For most drivers those two are fairly separate,
given that most modern drivers don't care about the SCSI "tagged" status
of a command at all, and many old drivers allow queuing of multiple
untagged commands in the driver.

Instead we start out with the ->simple_tags flag set before calling
->slave_configure, which is how all drivers actually looking at
->simple_tags except for one worke anyway.  The one other case looks
broken, but I've kept the behavior as-is for now.

Except for that we only change ->simple_tags from the ->change_queue_type,
and when rejecting a tag message in a single driver, so keeping this
churn out of scsi_adjust_queue_depth is a clear win.

Now that the usage of scsi_adjust_queue_depth is more obvious we can
also remove all the trivial instances in ->slave_alloc or ->slave_configure
that just set it to the cmd_per_lun default.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NMike Christie <michaelc@cs.wisc.edu>
Reviewed-by: NHannes Reinecke <hare@suse.de>
Reviewed-by: NMartin K. Petersen <martin.petersen@oracle.com>

Allow a driver to ask for block layer tags by setting .use_blk_tags in the
host template, in which case it will always see a valid value in
request->tag, similar to the behavior when using blk-mq.  This means even
SCSI "untagged" commands will now have a tag, which is especially useful
when using a host-wide tag map.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NMike Christie <michaelc@cs.wisc.edu>
Reviewed-by: NMartin K. Petersen <martin.petersen@oracle.com>
Reviewed-by: NHannes Reinecke <hare@suse.de>

Open-code scsi_print_result in sd.c, and cleanup logging to
not print duplicate informations.
Also remove the call to scsi_show_result() in ufshcd.c
to be consistent with other callers of scsi_execute().

With that we can remove scsi_show_result in constants.c
Signed-off-by: NHannes Reinecke <hare@suse.de>
Reviewed-by: NRobert Elliott <elliott@hp.com>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

We should be using the standard dev_printk() variants for
sense code printing.

[hch: remove __scsi_print_sense call in xen-scsiback, Acked by Juergen]
[hch: folded bracing fix from Dan Carpenter]
Signed-off-by: NHannes Reinecke <hare@suse.de>
Reviewed-by: NRobert Elliott <elliott@hp.com>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

- Adding some of the definitions missing in unipro.h, including power
  enumeration.
- Read Modify Write Line helper function
- Indication for the type of suspend
Signed-off-by: NDolev Raviv <draviv@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

Add capability to control the auto bkops during suspend.
If host explicitly enables the auto bkops (background operation) on device
then only device would perform the bkops on its own. If auto bkops is not
enabled explicitly and if the device reaches to state where it must do
background operation, device would raise the urgent bkops exception event
to host and then host will enable the auto bkops on device. This patch
adds the option to choose whether auto bkops should be enabled during
runtime suspend or not. Since we don't want to keep the device active to
perform the non critical bkops, host will enable urgent bkops only.

Keep auto-bkops enabled after resume if urgent bkops needed.
If device bkops status shows that its in critical need of executing
background operations, host should allow the device to continue doing
background operations.
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NDolev Raviv <draviv@codeaurora.org>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

The clocks for UFS device will be managed by generic DVFS (Dynamic
Voltage and Frequency Scaling) framework within kernel. This devfreq
framework works with different governors to scale the clocks. By default,
UFS devices uses simple_ondemand governor which scales the clocks up if
the load is more than upthreshold and scales down if the load is less than
downthreshold.
Signed-off-by: NSahitya Tummala <stummala@codeaurora.org>
Signed-off-by: NDolev Raviv <draviv@codeaurora.org>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

The UFS controller clocks can be gated after certain period of
inactivity, which is typically less than runtime suspend timeout.
In addition to clocks the link will also be put into Hibern8 mode
to save more power.

The clock gating can be turned on by enabling the capability
UFSHCD_CAP_CLK_GATING. To enable entering into Hibern8 mode as part of
clock gating, set the capability UFSHCD_CAP_HIBERN8_WITH_CLK_GATING.

The tracing events for clock gating can be enabled through debugfs as:
echo 1 > /sys/kernel/debug/tracing/events/ufs/ufshcd_clk_gating/enable
cat /sys/kernel/debug/tracing/trace_pipe
Signed-off-by: NSahitya Tummala <stummala@codeaurora.org>
Signed-off-by: NDolev Raviv <draviv@codeaurora.org>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

Sometimes, the device shall report its maximum power and speed
capabilities, but we might not wish to configure it to use those
maximum capabilities.
This change adds support for the vendor specific host driver to
implement power change notify callback.

To enable configuring different power modes (number of lanes,
gear number and fast/slow modes) it is necessary to split the
configuration stage from the stage that reads the device max power mode.
In addition, it is not required to read the configuration more than
once, thus the configuration is stored after reading it once.
Signed-off-by: NDolev Raviv <draviv@codeaurora.org>
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

This patch adds support for UFS device and UniPro link power management
during runtime/system PM.

Main idea is to define multiple UFS low power levels based on UFS device
and UFS link power states. This would allow any specific platform or pci
driver to choose the best suited low power level during runtime and
system suspend based on their power goals.

bkops handlig:
To put the UFS device in sleep state when bkops is disabled, first query
the bkops status from the device and enable bkops on device only if
device needs time to perform the bkops.

START_STOP handling:
Before sending START_STOP_UNIT to the device well-known logical unit
(w-lun) to make sure that the device w-lun unit attention condition is
cleared.

Write protection:
UFS device specification allows LUs to be write protected, either
permanently or power on write protected. If any LU is power on write
protected and if the card is power cycled (by powering off VCCQ and/or
VCC rails), LU's write protect status would be lost. So this means those
LUs can be written now. To ensures that UFS device is power cycled only
if the power on protect is not set for any of the LUs, check if power on
write protect is set and if device is in sleep/power-off state & link in
inactive state (Hibern8 or OFF state).
If none of the Logical Units on UFS device is power on write protected
then all UFS device power rails (VCC, VCCQ & VCCQ2) can be turned off if
UFS device is in power-off state and UFS link is in OFF state. But current
implementation would disable all device power rails even if UFS link is
not in OFF state.

Low power mode:
If UFS link is in OFF state then UFS host controller can be power collapsed
to avoid leakage current from it. Note that if UFS host controller is power
collapsed, full UFS reinitialization will be required on resume to
re-establish the link between host and device.
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NDolev Raviv <draviv@codeaurora.org>
Signed-off-by: NSujit Reddy Thumma <sthumma@codeaurora.org>
Signed-off-by: NChristoph Hellwig <hch@lst.de>