If UFS host device is in runtime-suspended state while UFS shutdown
callback is invoked, UFS device shall be resumed for register
accesses. Currently only UFS local runtime resume function will be invoked
to wake up the host.  This is not enough because if someone triggers
runtime resume from block layer, then race may happen between shutdown and
runtime resume flow, and finally lead to unlocked register access.

To fix this, in ufshcd_shutdown(), use pm_runtime_get_sync() instead of
resuming UFS device by ufshcd_runtime_resume() "internally" to let runtime
PM framework manage the whole resume flow.

Link: https://lore.kernel.org/r/20201119062916.12931-1-stanley.chu@mediatek.com
Fixes: 57d104c1 ("ufs: add UFS power management support")
Reviewed-by: NCan Guo <cang@codeaurora.org>
Signed-off-by: NStanley Chu <stanley.chu@mediatek.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

If someone plays with the UFS clk scaling devfreq governor through sysfs,
ufshcd_devfreq_scale may be called even when HBA is not runtime ACTIVE.
This can lead to unexpected error. We cannot just protect it by calling
pm_runtime_get_sync() because that may cause a race condition since HBA
runtime suspend ops need to suspend clk scaling. To fix this call
pm_runtime_get_noresume() and check HBA's runtime status. Only proceed if
HBA is runtime ACTIVE, otherwise just bail.

governor_store
 devfreq_performance_handler
  update_devfreq
   devfreq_set_target
    ufshcd_devfreq_target
     ufshcd_devfreq_scale

Link: https://lore.kernel.org/r/1600758548-28576-1-git-send-email-cang@codeaurora.orgReviewed-by: NStanley Chu <stanley.chu@mediatek.com>
Signed-off-by: NCan Guo <cang@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

WB-related sysfs entries can be accessed even when an UFS device does not
support the feature. The descriptors which are not supported by the UFS
device may be wrongly reported when they are accessed from their
corrsponding sysfs entries. Fix it by adding a sanity check of parameter
offset against the actual decriptor length.

Link: https://lore.kernel.org/r/1603346348-14149-1-git-send-email-cang@codeaurora.orgReviewed-by: NAsutosh Das <asutoshd@codeaurora.org>
Acked-by: NDaejun Park <daejun7.park@samsung.com>
Signed-off-by: NCan Guo <cang@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Add the missing destroy_workqueue() before return from ufshcd_init in the
error handling case as well as in ufshcd_remove.

Link: https://lore.kernel.org/r/20201110074223.41280-1-miaoqinglang@huawei.com
Fixes: 4db7a236 ("scsi: ufs: Fix concurrency of error handler and other error recovery paths")
Suggested-by: NAvri Altman <Avri.Altman@wdc.com>
Reviewed-by: NAsutosh Das <asutoshd@codeaurora.org>
Reviewed-by: NAvri Altman <avri.altman@wdc.com>
Signed-off-by: NQinglang Miao <miaoqinglang@huawei.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Use the uic_cmd->cmd_active as a flag to track the lifecycle of an UIC cmd.
The flag is set before sending the UIC cmd and cleared in IRQ handler. When
a PMC or UIC cmd completion timeout happens, if the flag is not set,
instead of returning timeout error, we still treat it as a successful
operation.  This is to deal with the scenario in which completion has been
raised but the one waiting for the completion cannot be awaken in time due
to kernel scheduling problem.

Link: https://lore.kernel.org/r/1604384682-15837-3-git-send-email-cang@codeaurora.orgReviewed-by: NStanley Chu <stanley.chu@mediatek.com>
Signed-off-by: NCan Guo <cang@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The scsi_block_reqs_cnt increased in ufshcd_hold() is supposed to be
decreased back in ufshcd_ungate_work() in a paired way. However, if
specific ufshcd_hold/release sequences are met, it is possible that
scsi_block_reqs_cnt is increased twice but only one ungate work is
queued. To make sure scsi_block_reqs_cnt is handled by ufshcd_hold() and
ufshcd_ungate_work() in a paired way, increase it only if queue_work()
returns true.

Link: https://lore.kernel.org/r/1604384682-15837-2-git-send-email-cang@codeaurora.orgReviewed-by: NHongwu Su <hongwus@codeaurora.org>
Reviewed-by: NStanley Chu <stanley.chu@mediatek.com>
Reviewed-by: NBean Huo <beanhuo@micron.com>
Signed-off-by: NCan Guo <cang@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Boot occasionally fails with some Samsung low-power UFS devices. The reason
is that these devices have a little bit higher latency for NOP OUT
responses. This causes boot to fail because the NOP OUT command is issued
during initialization to check whether the device transport protocol is
ready or not. Increase NOP_OUT_TIMEOUT value from 30 to 50ms.

Link: https://lore.kernel.org/r/231786897.01599016081767.JavaMail.epsvc@epcpadp2Acked-by: NAvri Altman <avri.altman@wdc.com>
Signed-off-by: NDaejun Park <daejun7.park@samsung.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Setting the Auto-Hibernate Timer to zero is a valid setting which indicates
the Auto-Hibernate feature being disabled. Correctly support this setting.

In addition, when the timer value is queried from sysfs, read from the host
controller's register and return that value instead of using the RAM value.

Link: https://lore.kernel.org/r/b141cfcd7998b8933635828b56fbb64f8ad4d175.1598661071.git.nguyenb@codeaurora.orgAcked-by: NStanley Chu <stanley.chu@mediatek.com>
Signed-off-by: NBao D. Nguyen <nguyenb@codeaurora.org>
Signed-off-by: NAsutosh Das <asutoshd@codeaurora.org>
Signed-off-by: NCan Guo <cang@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

PA Layer issues a LINERESET to the PHY at the recovery step in the Power
Mode change operation. If it happens during auto or manual hibern8 enter,
even if hibern8 enter succeeds, UFS power mode shall be set to PWM-G1 mode
and kept in that mode after exit from hibern8, leading to bad performance.
Handle the LINERESET in the eh_work by restoring power mode to HS mode
after all pending reqs and tasks are cleared from doorbell.

Link: https://lore.kernel.org/r/1598321228-21093-3-git-send-email-cang@codeaurora.orgSigned-off-by: NCan Guo <cang@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

To recover non-fatal errors, no full reset is required, err_handler only
clears those pending TRs/TMRs so that SCSI layer can re-issue them. In
current err_handler, TRs are directly cleared from UFS host's doorbell but
not aborted from device side. However, according to the UFSHCI JEDEC spec,
the host software shall use UTP Transfer Request List Clear Register to
clear a task from UFS host's doorbell only when a UTP Transfer Request is
expected to not be completed, e.g. when the host software receives a
“FUNCTION COMPLETE” Task Management response which means a Transfer Request
was aborted. To follow the UFSHCI JEDEC spec, in err_handler, abort one TR
before clearing it from doorbell.

Link: https://lore.kernel.org/r/1598321228-21093-2-git-send-email-cang@codeaurora.orgAcked-by: NStanley Chu <stanley.chu@mediatek.com>
Signed-off-by: NCan Guo <cang@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Fix ufshcd_print_trs() to consider UFSHCD_QUIRK_PRDT_BYTE_GRAN when using
utp_transfer_req_desc::prd_table_length, so that it doesn't treat the
number of bytes as the number of entries.

Originally from Kiwoong Kim
(https://lkml.kernel.org/r/20200218233115.8185-1-kwmad.kim@samsung.com).

Link: https://lore.kernel.org/r/20200826021040.152148-1-ebiggers@kernel.org
Fixes: 26f968d7 ("scsi: ufs: Introduce UFSHCD_QUIRK_PRDT_BYTE_GRAN quirk")
Cc: Alim Akhtar <alim.akhtar@samsung.com>
Cc: Kiwoong Kim <kwmad.kim@samsung.com>
Signed-off-by: NEric Biggers <ebiggers@google.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

We have two knobs to control flush for write booster,
fWriteBoosterBufferFlushDuringHibernate and fWriteBoosterBufferFlushEn.

Some vendors use only fWriteBoosterBufferFlushDuringHibernate because this
can reportedly cover most scenarios. Also, there have been some reports
that flush by fWriteBoosterBufferFlushEn could lead to increased power
consumption thanks to unexpected internal operations. Consequently, we need
a way to enable or disable fWriteBoosterEn operations. Add quirk to bypass
manual flush.

Link: https://lore.kernel.org/r/ffdb0eda30515809f0ad9ee936b26917ee9b4593.1598319701.git.kwmad.kim@samsung.comReviewed-by: NAvri Altman <avri.altman@wdc.com>
Signed-off-by: NKiwoong Kim <kwmad.kim@samsung.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Replace the existing /* fall through */ comments and its variants with
the new pseudo-keyword macro fallthrough[1]. Also, remove unnecessary
fall-through markings when it is the case.

[1] https://www.kernel.org/doc/html/v5.7/process/deprecated.html?highlight=fallthrough#implicit-switch-case-fall-throughSigned-off-by: NGustavo A. R. Silva <gustavoars@kernel.org>

Commit 5586dd8e ("scsi: ufs: Fix a race condition between error handler
and runtime PM ops") moves the ufshcd_scsi_block_requests() inside
err_handler() but forgets to remove the ufshcd_scsi_unblock_requests() in
the early return path. Correct the mistake.

Link: https://lore.kernel.org/r/1597798958-24322-1-git-send-email-cang@codeaurora.org
Fixes: 5586dd8e ("scsi: ufs: Fix a race condition between error handler and runtime PM ops")
Reviewed-by: NAsutosh Das <asutoshd@codeaurora.org>
Reviewed-by: Hongwu Su<hongwus@codeaurora.org>
Signed-off-by: NCan Guo <cang@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Currently, the UFS driver busy waits for fDeviceInit to be cleared. Provide
an upper bound and sleep between attempts instead of busy waiting.

Link: https://lore.kernel.org/r/1597053747-75171-1-git-send-email-kwmad.kim@samsung.comTested-by: NKiwoong Kim <kwmad.kim@samsung.com>
Reviewed-by: NAvri Altman <avri.altman@wdc.com>
Signed-off-by: NKiwoong Kim <kwmad.kim@samsung.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Link: https://lore.kernel.org/r/20200814095034.20709-3-huobean@gmail.comReviewed-by: NStanley Chu <stanley.chu@mediatek.com>
Reviewed-by: NAsutosh Das <asutoshd@codeaurora.org>
Signed-off-by: NBean Huo <beanhuo@micron.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

ufshcd_comp_devman_upiu() was poorly named leading people to think it was a
completion function. Rename it to ufshcd_compose_devman_upiu().

Link: https://lore.kernel.org/r/20200814095034.20709-2-huobean@gmail.comReviewed-by: NStanley Chu <stanley.chu@mediatek.com>
Reviewed-by: NAsutosh Das <asutoshd@codeaurora.org>
Acked-by: NAvri Altman <avri.altman@wdc.com>
Signed-off-by: NBean Huo <beanhuo@micron.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

In current UFS task abort hook, namely ufshcd_abort(), if one task is
aborted successfully, clk_gating.active_reqs held by this task is not
decreased, which makes clk_gating.active_reqs stay above zero forever, thus
clock gating would never happen. Instead of releasing resources of one task
"manually", use the existing func __ufshcd_transfer_req_compl().  This
change also eliminates a possible race of scsi_dma_unmap() from the real
completion in IRQ handler path.

Link: https://lore.kernel.org/r/1596975355-39813-10-git-send-email-cang@codeaurora.org
Fixes: 1ab27c9c ("ufs: Add support for clock gating")
CC: Stanley Chu <stanley.chu@mediatek.com>
Reviewed-by: NStanley Chu <stanley.chu@mediatek.com>
Reviewed-by: NAsutosh Das <asutoshd@codeaurora.org>
Signed-off-by: NCan Guo <cang@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

If the bit corresponding to a task in the Doorbell register has been
cleared, no need to poll the status of the task on the device side and to
send an Abort Task TM. Instead, let it directly goto cleanup.

In addition, to keep original debug output, move the goto below the debug
print.

Link: https://lore.kernel.org/r/20200811141859.27399-3-huobean@gmail.comReviewed-by: NStanley Chu <stanley.chu@mediatek.com>
Reviewed-by: NCan Guo <cang@codeaurora.org>
Signed-off-by: NBean Huo <beanhuo@micron.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

If somehow no interrupt notification is raised for a completed request and
its doorbell bit is cleared by host, UFS driver needs to cleanup its
outstanding bit in ufshcd_abort(). Otherwise, system may behave abnormally
in the following scenario:

After ufshcd_abort() returns, this request will be requeued by SCSI layer
with its outstanding bit set. Any future completed request will trigger
ufshcd_transfer_req_compl() to handle all "completed outstanding bits". At
this time the "abnormal outstanding bit" will be detected and the "requeued
request" will be chosen to execute request post-processing flow. This is
wrong because this request is still "alive".

Link: https://lore.kernel.org/r/20200811141859.27399-2-huobean@gmail.comReviewed-by: NCan Guo <cang@codeaurora.org>
Acked-by: NAvri Altman <avri.altman@wdc.com>
Signed-off-by: NStanley Chu <stanley.chu@mediatek.com>
Signed-off-by: NBean Huo <beanhuo@micron.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

For shared interrupts, the interrupt status might be zero, so check that
first.

Link: https://lore.kernel.org/r/20200811133936.19171-2-adrian.hunter@intel.comReviewed-by: NAvri Altman <avri.altman@wdc.com>
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The interrupt might be shared, in which case it is not an error for the
interrupt handler to be called when the interrupt status is zero, so don't
print the message unless there was enabled interrupt status.

Link: https://lore.kernel.org/r/20200811133936.19171-1-adrian.hunter@intel.com
Fixes: 9333d775 ("scsi: ufs: Fix irq return code")
Reviewed-by: NAvri Altman <avri.altman@wdc.com>
Signed-off-by: NAdrian Hunter <adrian.hunter@intel.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

In ufshcd_suspend(), after clk-gating is suspended and link is set
as Hibern8 state, ufshcd_hold() is still possibly invoked before
ufshcd_suspend() returns. For example, MediaTek's suspend vops may
issue UIC commands which would call ufshcd_hold() during the command
issuing flow.

Now if UFSHCD_CAP_HIBERN8_WITH_CLK_GATING capability is enabled,
then ufshcd_hold() may enter infinite loops because there is no
clk-ungating work scheduled or pending. In this case, ufshcd_hold()
shall just bypass, and keep the link as Hibern8 state.

Link: https://lore.kernel.org/r/20200809050734.18740-1-stanley.chu@mediatek.comReviewed-by: NAvri Altman <avri.altman@wdc.com>
Co-developed-by: NAndy Teng <andy.teng@mediatek.com>
Signed-off-by: NAndy Teng <andy.teng@mediatek.com>
Signed-off-by: NStanley Chu <stanley.chu@mediatek.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The current IRQ handler blocks SCSI requests before scheduling eh_work,
when error handler calls pm_runtime_get_sync, if ufshcd_suspend/resume
sends a SCSI cmd, most likely the SSU cmd, since SCSI requests are blocked,
pm_runtime_get_sync() will never return because ufshcd_suspend/resume is
blocked by the SCSI cmd.

 - In queuecommand path, hba->ufshcd_state check and ufshcd_send_command
   should stay under the same spin lock. This is to make sure that no more
   commands leak into doorbell after hba->ufshcd_state is changed.

 - Don't block SCSI requests before error handler starts to run, let error
   handler block SCSI requests when it is ready to start error recovery.

 - Don't let SCSI layer keep requeuing the SCSI cmds sent from HBA runtime
   PM ops, let them pass or fail them. Let them pass if eh_work is
   scheduled due to non-fatal errors. Fail them if eh_work is scheduled due
   to fatal errors, otherwise the cmds may eventually time out since UFS is
   in bad state, which gets error handler blocked for too long. If we fail
   the SCSI cmds sent from HBA runtime PM ops, HBA runtime PM ops fails
   too, but it does not hurt since error handler can recover HBA runtime PM
   error.

Link: https://lore.kernel.org/r/1596975355-39813-9-git-send-email-cang@codeaurora.orgReviewed-by: NBean Huo <beanhuo@micron.com>
Reviewed-by: NAsutosh Das <asutoshd@codeaurora.org>
Signed-off-by: NCan Guo <cang@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Performing dumps in the IRQ handler causes system stability issues. Move
dumps to the error handler and only print basic host registers here.

Link: https://lore.kernel.org/r/1596975355-39813-8-git-send-email-cang@codeaurora.orgReviewed-by: NBean Huo <beanhuo@micron.com>
Reviewed-by: NAsutosh Das <asutoshd@codeaurora.org>
Signed-off-by: NCan Guo <cang@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The current error handler can not recover HBA runtime PM error if
ufshcd_suspend/resume has failed due to UFS errors, e.g. hibern8 enter/exit
error or SSU cmd error. When this happens, error handler may fail
performing a full reset and restore because error handler always assumes
that power, IRQs and clocks are ready after pm_runtime_get_sync returns,
but actually they are not if ufshcd_resume fails[1].

If ufschd_suspend/resume fails due to UFS errors, runtime PM framework
saves the error value to dev.power.runtime_error. After that, HBA dev
runtime suspend/resume would not be invoked anymore unless runtime_error is
cleared[2].

In case of ufshcd_suspend/resume fails due to UFS errors, for scenario [1],
error handler cannot assume anything of pm_runtime_get_sync, meaning error
handler should explicitly turn ON powers, IRQs and clocks again. To get the
HBA runtime PM work as regard for scenario [2], error handler can clear the
runtime_error by calling pm_runtime_set_active() if full reset and restore
succeeds. And, more important, if pm_runtime_set_active() returns no error,
which means runtime_error has been cleared, we also need to resume those
scsi devices under HBA in case any of them has failed to be resumed due to
HBA runtime resume failure. This is to unblock blk_queue_enter in case
there are bios waiting inside it.

Link: https://lore.kernel.org/r/1596975355-39813-7-git-send-email-cang@codeaurora.orgReviewed-by: NBean Huo <beanhuo@micron.com>
Reviewed-by: NAsutosh Das <asutoshd@codeaurora.org>
Signed-off-by: NCan Guo <cang@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Error recovery can be invoked from multiple code paths, including hibern8
enter/exit (from ufshcd_link_recovery), ufshcd_eh_host_reset_handler() and
eh_work scheduled from IRQ context. Ultimately, these paths are all trying
to invoke ufshcd_reset_and_restore() in either a synchronous or
asynchronous manner. This causes problems:

 - If link recovery happens during ungate work, ufshcd_hold() would be
   called recursively. Although commit 53c12d0e ("scsi: ufs: fix error
   recovery after the hibern8 exit failure") fixed a deadlock due to
   recursive calls of ufshcd_hold() by adding a check of eh_in_progress
   into ufshcd_hold, this check allows eh_work to run in parallel while
   link recovery is running.

 - Similar concurrency can also happen when error recovery is invoked from
   ufshcd_eh_host_reset_handler and ufshcd_link_recovery.

 - Concurrency can even happen between eh_works. eh_work, currently queued
   on system_wq, is allowed to have multiple instances running in parallel,
   but we don't have proper protection for that.

If any of above concurrency scenarios happen, error recovery would fail and
lead ufs device and host into bad states. To fix the concurrency problem,
this change queues eh_work on a single threaded workqueue and removes link
recovery calls from the hibern8 enter/exit path. In addition, make use of
eh_work in eh_host_reset_handler instead of calling
ufshcd_reset_and_restore. This unifies the UFS error recovery mechanism.

According to the UFSHCI JEDEC spec, hibern8 enter/exit error occurs when
the link is broken. This essentially applies to any power mode change
operations (since they all use PACP_PWR cmds in UniPro layer). So, if a
power mode change operation (including AH8 enter/exit) fails, mark link
state as UIC_LINK_BROKEN_STATE and schedule the eh_work.  In this case,
error handler needs to do a full reset and restore to recover the link back
to active. Before the link state is recovered to active,
ufshcd_uic_pwr_ctrl simply returns -ENOLINK to avoid more errors.

Link: https://lore.kernel.org/r/1596975355-39813-6-git-send-email-cang@codeaurora.orgReviewed-by: NBean Huo <beanhuo@micron.com>
Reviewed-by: NAsutosh Das <asutoshd@codeaurora.org>
Signed-off-by: NCan Guo <cang@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Information about the last interrupt status and timestamp is helpful when
debugging system stability issues (IRQ starvation, for instance). Add this
information to ufshcd_print_host_state() output.

In addition, UFS device information such as model name and firmware version
also comes in handy during debugging. This is printed as well.

Link: https://lore.kernel.org/r/1596975355-39813-5-git-send-email-cang@codeaurora.orgReviewed-by: NAvri Altman <avri.altman@wdc.com>
Reviewed-by: NHongwu Su <hongwus@codeaurora.org>
Reviewed-by: NAsutosh Das <asutoshd@codeaurora.org>
Reviewed-by: NStanley Chu <stanley.chu@mediatek.com>
Reviewed-by: NBean Huo <beanhuo@micron.com>
Signed-off-by: NCan Guo <cang@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Clock gating can be turned on/off selectively which means the associated
state information is only correct if the feature is enabled. This change
makes sure that we only look at state of clk-gating if it is enabled.

Link: https://lore.kernel.org/r/1596975355-39813-2-git-send-email-cang@codeaurora.orgReviewed-by: NAvri Altman <avri.altman@wdc.com>
Reviewed-by: NHongwu Su <hongwus@codeaurora.org>
Reviewed-by: NStanley Chu <stanley.chu@mediatek.com>
Reviewed-by: NBean Huo <beanhuo@micron.com>
Reviewed-by: NAsutosh Das <asutoshd@codeaurora.org>
Signed-off-by: NCan Guo <cang@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Some UFS devices require delay after VCC power rail is turned off.
Introduce a device quirk "DELAY_AFTER_LPM" to add 5ms delay after VCC
power-off during suspend flow.

Link: https://lore.kernel.org/r/20200729051840.31318-2-stanley.chu@mediatek.comReviewed-by: NAndy Teng <andy.teng@mediatek.com>
Reviewed-by: NPeter Wang <peter.wang@mediatek.com>
Reviewed-by: NCan Guo <cang@codeaurora.org>
Acked-by: NAvri Altman <Avri.Altman@wdc.com>
Signed-off-by: NStanley Chu <stanley.chu@mediatek.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Add missing setup_xfer_req() call in ufshcd_issue_devman_upiu_cmd() in
ufs-bsg path. Relocate existing setup_xfer_req() calls to a common place,
i.e., ufshcd_send_command(), to simplify the driver.

Link: https://lore.kernel.org/r/20200706060707.32608-3-stanley.chu@mediatek.comAcked-by: NAvri Altman <Avri.Altman@wdc.com>
Signed-off-by: NStanley Chu <stanley.chu@mediatek.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Simplify recording command completion time in __ufshcd_transfer_req_compl()
by assigning lrbp->compl_time_stamp in a unified location.

Link: https://lore.kernel.org/r/20200706060707.32608-2-stanley.chu@mediatek.comAcked-by: NAvri Altman <Avri.Altman@wdc.com>
Signed-off-by: NStanley Chu <stanley.chu@mediatek.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Wire up ufshcd.c with the UFS Crypto API, the block layer inline encryption
additions and the keyslot manager.

Many existing inline crypto devices require some additional behaviour not
specified in the UFSHCI v2.1 specification - as such the vendor specific
drivers will need to be updated where necessary to make it possible to use
those devices. Some of these changes have already been proposed upstream,
such as for the Qualcomm 845 SoC at
https://lkml.kernel.org/linux-scsi/20200501045111.665881-1-ebiggers@kernel.org/
and for ufs-mediatek at
https://lkml.kernel.org/linux-scsi/20200304022101.14165-1-stanley.chu@mediatek.com/

This patch has been tested on the db845c, sm8150-mtp and sm8250-mtp
(which have Qualcomm chipsets) and on some mediatek chipsets using these
aforementioned vendor specific driver updates.

Link: https://lore.kernel.org/r/20200706200414.2027450-4-satyat@google.comReviewed-by: NEric Biggers <ebiggers@google.com>
Reviewed-by: NStanley Chu <stanley.chu@mediatek.com>
Reviewed-by: NAlim Akhtar <alim.akhtar@samsung.com>
Signed-off-by: NSatya Tangirala <satyat@google.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Add the crypto registers and structs defined in v2.1 of the JEDEC UFSHCI
specification in preparation to add support for inline encryption to UFS.

Link: https://lore.kernel.org/r/20200706200414.2027450-2-satyat@google.comReviewed-by: NEric Biggers <ebiggers@google.com>
Reviewed-by: NStanley Chu <stanley.chu@mediatek.com>
Reviewed-by: NAlim Akhtar <alim.akhtar@samsung.com>
Signed-off-by: NSatya Tangirala <satyat@google.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

According to the UFS Spec, the Flags field in the UPIU is one byte in size,
not 4. Change it to be u8.

Link: https://lore.kernel.org/r/20200706123936.24799-1-huobean@gmail.comTested-by: NAlim Akhtar <alim.akhtar@samsung.com>
Reviewed-by: NAlim Akhtar <alim.akhtar@samsung.com>
Reviewed-by: NAvri Altman <avri.altman@wdc.com>
Signed-off-by: NBean Huo <beanhuo@micron.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

If a UFS device is not qualified to use WriteBooster, either due to wrong
UFS version or device-specific quirks, then the capability in host shall be
disabled to prevent any WriteBooster operations in the future.

Link: https://lore.kernel.org/r/20200625030430.25048-1-stanley.chu@mediatek.com
Fixes: 3d17b9b5 ("scsi: ufs: Add write booster feature support")
Tested-by: NSteev Klimaszewski <steev@kali.org>
Reviewed-by: NAvri Altman <avri.altman@wdc.com>
Reviewed-by: NBean Huo <beanhuo@micron.com>
Signed-off-by: NStanley Chu <stanley.chu@mediatek.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The UFS load calculation is based on "total_time" and "busy_time" in a
devfreq window.  However, the source of time is different for both
parameters: "busy_time" is assigned from "jiffies" thus has different
accuracy from "total_time" which is assigned from ktime_get().

In addition, the time of window boundary is not exactly the same as the
starting busy time in this window if UFS is actually busy in the beginning
of the window. A similar accuracy error may also happen for the end of busy
time in current window.

To guarantee the precision of load calculation, we need to

1. Align time accuracy of both devfreq_dev_status.total_time and
   devfreq_dev_status.busy_time. For example, use "ktime_get()" directly.

2. Align the following timelines:
   - The beginning time of devfreq windows
   - The beginning of busy time in a new window
   - The end of busy time in the current window

Link: https://lore.kernel.org/r/20200611101043.6379-1-stanley.chu@mediatek.com
Fixes: a3cd5ec5 ("scsi: ufs: add load based scaling of UFS gear")
Reviewed-by: NAvri Altman <avri.altman@wdc.com>
Signed-off-by: NStanley Chu <stanley.chu@mediatek.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Use the ftrace infrastructure to conditionally trace UFS UIC command
events.

New trace event "ufshcd_uic_command" is created, which samples the
following UFS UIC command data:

 - Device name
 - Optional identification string
 - UIC command opcode
 - UIC command argument1
 - UIC command argument2
 - UIC command argement3

Usage:
	echo 1 > /sys/kernel/debug/tracing/events/ufs/enable
	cat /sys/kernel/debug/tracing/trace_pipe

Link: https://lore.kernel.org/r/20200615072235.23042-3-stanley.chu@mediatek.comAcked-by: NAvri Altman <avri.altman@wdc.com>
Signed-off-by: NStanley Chu <stanley.chu@mediatek.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Clean up the following items:

 - Sort vendor names in alphabetical order

 - Squash quirks as compact as possible in device table to enhance
   performance of the lookup

 - Sort device quirks in alphabetical order

Link: https://lore.kernel.org/r/20200612012625.6615-3-stanley.chu@mediatek.comReviewed-by: NAlim Akhtar <alim.akhtar@samsung.com>
Signed-off-by: NStanley Chu <stanley.chu@mediatek.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

It is confirmed that Micron device needs DELAY_BEFORE_LPM quirk to have a
delay before VCC is powered off. Sdd Micron vendor ID and this quirk for
Micron devices.

Link: https://lore.kernel.org/r/20200612012625.6615-2-stanley.chu@mediatek.comReviewed-by: NBean Huo <beanhuo@micron.com>
Reviewed-by: NAlim Akhtar <alim.akhtar@samsung.com>
Signed-off-by: NStanley Chu <stanley.chu@mediatek.com>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>