This patch adds the profiling support for some of the time critical
operations like hibern8 enter/exit, clock gating & clock scaling.
Reviewed-by: NVenkat Gopalakrishnan <venkatg@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Immediately after successful UFS link startup, UFS link power mode would
be in PWM-G1, 1-lane, SLOW-AUTO mode. But currently we are doing few
of the DME local/peer attributes access before setting the "hba->pwr_info"
to default power mode. If we are doing link startup as part of error
recovery then old power mode might be set to FAST mode and doing DME peer
access (after link startup but before updating "hba->pwr_info" to default
power mode) unintentionally tries to switch from FAST to FAST_AUTO mode (if
UFSHCD_QUIRK_DME_PEER_ACCESS_AUTO_MODE quirk is enabled).

Above issue is fixed by setting the default power mode immediately after
successful link startup.
Reviewed-by: NSahitya Tummala <stummala@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

UFS device requires to perform bkops (back ground operations) periodically
but host can control (via auto-bkops parameter of device) when device can
perform bkops based on its performance requirements. In general, host
would like to enable the device's auto-bkops only when it's not doing any
regular data transfer but sometimes device may not behave properly if host
keeps the auto-bkops disabled. This change adds the capability to let the
device auto-bkops always enabled except suspend.
Reviewed-by: NSahitya Tummala <stummala@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

UFS device and link can be put in multiple different low power modes hence
UFS driver supports multiple different low power modes.
This change sets the default UFS power management level which should put
the link hibernate state and device in sleep state. This default power
management level gives good  power savings with relatively less enter/exit
latencies.
Reviewed-by: NYaniv Gardi <ygardi@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

This patch provides the sysfs attribute to choose the power management
level for UFS runtime and system suspend.
Reviewed-by: NSujit Reddy Thumma <sthumma@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Provide an option to enable/disable clock scaling during runtime.
Write 1/0 to "clkscale_enable" sysfs node to enable/disable clock
scaling.
Signed-off-by: NSahitya Tummala <stummala@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Provide an option to enable/disable clock gating during runtime.
Write 1 or 0 to "clkgate_enable" sysfs node to enable/disable
clock gating.
Signed-off-by: NSahitya Tummala <stummala@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

When a command to a W-LU is timed out via scsi, error handling
will treat it as any other LU and send commands such as
START_STOP with wrong format or task abort. Those commands are
illegal for W-LU according to the UFS spec.
To solve it, when an error is recognized those steps are skipped
and the last step, reset and restore process, is initiated.
Signed-off-by: NDolev Raviv <draviv@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

This change adds the ftrace support for following:
1. UFS initialization time
2. Clock gating states
3. Clock scaling states
4. Power management APIs latency
5. BKOPs enable/disable

Usage:
	echo 1 > /sys/kernel/debug/tracing/events/ufs/enable
	cat /sys/kernel/debug/tracing/trace_pipe
Reviewed-by: NSahitya Tummala <stummala@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Inserts driver dumps for UFS Host Controller registers, Transfer Requests
and Task Management Requests.
The dumps will occur on driver initialization failure, ufshcd_abort() and
on error handling path.
Signed-off-by: NDolev Raviv <draviv@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

ktime_set(S,N) was required for the timespec storage type and is still
useful for situations where a Seconds and Nanoseconds part of a time value
needs to be converted. For anything where the Seconds argument is 0, this
is pointless and can be replaced with a simple assignment.
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>

The maximum value PA_SaveConfigTime is 250 (10us) but this is not enough
for some vendors. Gear switch from PWM to HS may fail even with this
max.  PA_SaveConfigTime. Gear switch can be issued by host controller as
an error recovery and any software delay will not help on this case so
we need to increase PA_SaveConfigTime to >32us as per vendor
recommendation.  This change adds a quirk to increase the
PA_SaveConfigTime parameter.
Reviewed-by: NVenkat Gopalakrishnan <venkatg@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The UFS HCI v2.1 includes a few additional registers. This change
updates the HCI register, the UFS version register content and the
Interrupt Status register.
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

We would by default like to run in FAST/SLOW mode instead
of FASTAUTO/SLOWAUTO mode for performance reasons. This
change sets the default speed mode to FAST/SLOW mode.
Reviewed-by: NVenkat Gopalakrishnan <venkatg@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Consider following sequence of events:
1. UFS is runtime suspended, link_state = Hibern8, device_state = sleep
2. System goes into system suspend, ufshcd_system_suspend() brings both
   link and device to active state and then puts the device in Power_Down
   state and link in OFF state.
3. System resumes at some later point in time, ufshcd_system_resume()
   doesn't do anything as UFS state is runtime suspended. Note that link
   is still on OFF state and device is in Power_Down state.
4. Now system again goes into suspend without any UFS accesses before it.
   ufshcd_system_suspend() again brings both link and device to active
   state and then puts the device in Power_Down state and link if OFF
   state. But it's unnecessary to bring the link & device in active state
   as both link and device are already in desired low power states. This
   change fixes this issue by adding proper state checks in
   ufshcd_system_suspend().
Reviewed-by: NGilad Broner <gbroner@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The condition in which error message is printed out was incorrect and
resulted error message only if retries exhausted.
But retries happens only if DME command is a peer command, and thus
DME commands which are not peer commands and fail are not printed out.
This change fixes this issue.
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The PHY_ADAPTER_ERROR status register indicates PHY lane errors
reported by the M-PHY layer. In some occasions the controller
can recover from such errors. When the error is not recoverable,
a stuck DB error will occur. Since the stuck DB error is spotted
separately, no action other than clearing the register is necessary.
Signed-off-by: NDolev Raviv <draviv@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

If we issue the link startup to the device while its UniPro state is
LinkDown (and device state is sleep/power-down) then link startup
will not move the device state to Active. Device will only move to
active state if the link starup is issued when its UniPro state is
LinkUp. So in this case, we would have to issue the link startup 2
times to make sure that device moves to active state.
Reviewed-by: NGilad Broner <gbroner@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Some UFS devices require host PA_TACTIVATE to be higher than
device PA_TACTIVATE otherwise it may get stuck during hibern8 sequence.
This change allows this by using quirk.
Reviewed-by: NVenkat Gopalakrishnan <venkatg@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

It is found thats UFS device may take longer than 30ms to respond to
query requests and in this case we might run into following scenario:

1. UFS host SW sends a query request to UFS device to read an attribute
   value. SW uses tag #31 for this purpose.
2. UFS host SW waits for 30ms to get the query response (and doorbell
   to be cleared by UFS host HW).
3. UFS device doesn't respond back within 30ms hence UFS host SW times
   out waiting for the query response.
4. UFS host SW clears the tag#31 from UTRLCLR register.
5. UFS host SW waits until UFS host HW to clear tag#31 from the doorbell
   register.
6. UFS host SW retries the same query request on same tag#31 (sends a query
   request to device to read an attribute value).
7. UFS host HW gets the query response from the device but this was
   intended as a query response for the 1st query request sent (step-1).
8. Now UFS device sends another query response to host (for query request
   sent @step-6).

Now there are 2 issues that could happen with above scenario:
1. UFS device should have actually responded back with only one query
   response but it is found that device may respond back with 2 query
   responses.
2. If UFS device responds back with 2 resposes on same tag, host HW/SW
   behaviour isn't predictable.

To avoid running into above scenario, we would basically allow device
to take longer (upto 1.5 seconds) for query response.
Reviewed-by: NGilad Broner <gbroner@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

While reading variable size descriptors (like string descriptor), some UFS
devices may report the "LENGTH" (field in "Transaction Specific fields" of
Query Response UPIU) same as what was requested in Query Request UPIU
instead of reporting the actual size of the variable size descriptor.
Although it's safe to ignore the "LENGTH" field for variable size
descriptors as we can always derive the length of the descriptor from
the descriptor header fields. Hence this change impose the length match
check only for fixed size descriptors (for which we always request the
correct size as part of Query Request UPIU).
Reviewed-by: NVenkat Gopalakrishnan <venkatg@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

When sending query to the device, the index  of the failure
is additional useful information that should be printed out as it
might specify the logical unit (LU) where the error occurred.
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Some of the queries might fail during init. To avoid
system failure, we add retry mechanism to issue queries
several times.
Signed-off-by: NDolev Raviv <draviv@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Some UFS host controllers may think granularities of PRDT length and
offset as bytes, not double words.
Signed-off-by: NKiwoong Kim <kwmad.kim@samsung.com>
Reviewed-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Add a new ufshcd_state, indicats that an err handler may get to run
immediately. Use UFSHCD_STATE_ERROR here looks not literaly correct.
Signed-off-by: NZang Leigang <zangleigang@hisilicon.com>
Reviewed-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Some UFS host controller may need to configure some things around
hibern8 enter/exit
Signed-off-by: NKiwoong Kim <kwmad.kim@samsung.com>
Reviewed-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Some UFS host controller may need to configure some things before any
task management request is issued
Signed-off-by: NKiwoong Kim <kwmad.kim@samsung.com>
Reviewed-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Some UFS host controller may need to configure some things before any
transfer request is issued.
Signed-off-by: NKiwoong Kim <kwmad.kim@samsung.com>
Reviewed-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

This patch uses the resource-managed to add the devfreq device.  This
function will make it easy to handle the devfreq device.

- struct devfreq *devm_devfreq_add_device(struct device *dev,
				  struct devfreq_dev_profile *profile,
				  const char *governor_name,
				  void *data);
Cc: Vinayak Holikatti <vinholikatti@gmail.com>
Cc: James E.J. Bottomley <jejb@linux.vnet.ibm.com>
Cc: Martin K. Petersen <martin.petersen@oracle.com>
Cc: linux-scsi@vger.kernel.org
Signed-off-by: NChanwoo Choi <cw00.choi@samsung.com>
Reviewed-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

When we are resuming the UFS device rails in HPM mode, we are first
powering on the VCC rail while VCCQ and VCCQ2 rails still being in LPM
mode. Some UFS devices may take VCC on event as hint that host wants UFS
device to be resumed and may start drawing more power from the
VCCQ/VCCQ2 rails (while they are still in LPM mode) causing voltage drop
on these rails. This change fixes this issue by bringing VCCQ & VCCQ2
rails out of LPM before powering on VCC rail.
Reviewed-by: NVenkat Gopalakrishnan <venkatg@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Currently clock scaling is suspended only after the host and device are
put in low power mode but we should avoid clock scaling running after
UFS link is put in low power mode (hibern8). This change suspends clock
scaling before putting host/device in low power mode.
Reviewed-by: NSahitya Tummala <stummala@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

If ufshcd pltfrm/pci driver's probe fails for some reason then ensure
that scsi host is released to avoid memory leak but managed memory
allocations (via devm_* calls) need not to be freed explicitly on probe
failure as memory allocated with these functions is automatically freed
on driver detach.
Reviewed-by: NSahitya Tummala <stummala@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

UFS devfreq clock scaling work may require clocks to be ON if it need to
execute some UFS commands hence it may request for clock hold before
issuing the command. But if UFS clock gating work is already running in
parallel, ungate work would end up waiting for the clock gating work to
finish and as clock gating work would also wait for the clock scaling
work to finish, we would enter in deadlock state. Here is the call trace
during this deadlock state:

Workqueue: devfreq_wq devfreq_monitor
	__switch_to
	__schedule
	schedule
	schedule_timeout
	wait_for_common
	wait_for_completion
	flush_work
	ufshcd_hold
	ufshcd_send_uic_cmd
	ufshcd_dme_get_attr
	ufs_qcom_set_dme_vs_core_clk_ctrl_clear_div
	ufs_qcom_clk_scale_notify
	ufshcd_scale_clks
	ufshcd_devfreq_target
	update_devfreq
	devfreq_monitor
	process_one_work
	worker_thread
	kthread
	ret_from_fork

Workqueue: events ufshcd_gate_work
	__switch_to
	__schedule
	schedule
	schedule_preempt_disabled
	__mutex_lock_slowpath
	mutex_lock
	devfreq_monitor_suspend
	devfreq_simple_ondemand_handler
	devfreq_suspend_device
	ufshcd_gate_work
	process_one_work
	worker_thread
	kthread
	ret_from_fork

Workqueue: events ufshcd_ungate_work
	__switch_to
	__schedule
	schedule
	schedule_timeout
	wait_for_common
	wait_for_completion
	flush_work
	__cancel_work_timer
	cancel_delayed_work_sync
	ufshcd_ungate_work
	process_one_work
	worker_thread
	kthread
	ret_from_fork

This change fixes this deadlock by doing this in devfreq work (devfreq_wq):
Try cancelling clock gating work. If we are able to cancel gating work
or it wasn't scheduled, hold the clock reference count until scaling is
in progress. If gate work is already running in parallel, let's skip
the frequecy scaling at this time and it will be retried once next scaling
window expires.
Reviewed-by: NSahitya Tummala <stummala@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

In a case where gate work is called as part of cancel work from ungate
path the clk state would be marked as REQ_CLKS_ON.  There is no point
gating the clocks and then end up turning them ON immediately in ungate
work, save time by skipping the gate work and change the clk state to
CLKS_ON as they are not turned off yet.
Signed-off-by: NVenkat Gopalakrishnan <venkatg@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

The ungate work turns on the clock before it exits hibern8, if the link
was put in hibern8 during clock gating work.  There occurs a race
condition when clock scaling work calls ufshcd_hold() to make sure low
power states cannot be entered, but that returns by checking only
whether the clocks are on.  This causes the clock scaling work to issue
UIC commands when the link is in hibern8 causing failures. Make sure we
exit hibern8 state before returning from ufshcd_hold().

Callstacks for race condition:

 ufshcd_scale_gear
 ufshcd_devfreq_scale
 ufshcd_devfreq_target
 update_devfreq
 devfreq_monitor
 process_one_work
 worker_thread
 kthread
 ret_from_fork

 ufshcd_uic_hibern8_exit
 ufshcd_ungate_work
 process_one_work
 worker_thread
 kthread
 ret_from_fork
Signed-off-by: NVenkat Gopalakrishnan <venkatg@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

In case the power configuration fails, skip further processing of the
probing function and return immediately. This has 2 reasons:

1. Don't allow the UFS to continue running in PWM

2. Avoid multiple calls to pm_runtime_put_sync() when not in error
   handling or power management contexts
Signed-off-by: NDov Levenglick <dovl@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

During runtime resume operation, clock scaling may get indirectly
resumed via call to ufshcd_set_dev_pwr_mode(): Start/Stop Unit command
times out and SCSI error handling ultimately calls the host reset
handler to recover, during which clock scaling is resumed.  Error case
exit path of runtime resume will disable clocks.  As clock scaling was
already resumed, it will get scheduled later on and try to access UFS
registers while clocks are disabled, resulting in unclocked register
access.
Signed-off-by: NGilad Broner <gbroner@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

According to UFS device specification, sense data can be only 18 bytes
long, this change makes the changes accordingly.
Signed-off-by: NGilad Broner <gbroner@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

Add a write memory barrier to make sure descriptors prepared are
actually written to memory before ringing the doorbell. We have also
added the write memory barrier after ringing the doorbell register so
that controller sees the new request immediately.
Signed-off-by: NGilad Broner <gbroner@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>

In this change there are a few fixes of possible NULL pointer access and
possible access to index that exceeds array boundaries.
Signed-off-by: NYaniv Gardi <ygardi@codeaurora.org>
Signed-off-by: NSubhash Jadavani <subhashj@codeaurora.org>
Signed-off-by: NMartin K. Petersen <martin.petersen@oracle.com>