Now that all the infrastructure is in place to support multiple required
OPPs, lets switch over to using it.

A new internal routine _set_required_opps() takes care of updating
performance state for all the required OPPs. With this the performance
state updates are supported even when the end device needs to configure
regulators as well, that wasn't the case earlier.

The pstates were earlier stored in the end device's OPP structures, that
also changes now as those values are stored in the genpd's OPP
structures. And so we switch over to using
pm_genpd_opp_to_performance_state() instead of
of_genpd_opp_to_performance_state() to get performance state for the
genpd OPPs.

The routine _generic_set_opp_domain() is not required anymore and is
removed.

On errors we don't try to recover by reverting to old settings as things
are really complex now and the calls here should never really fail
unless there is a bug. There is no point increasing the complexity, for
code which will never be executed.
Reviewed-by: NUlf Hansson <ulf.hansson@linaro.org>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

Multiple generic power domains for a consumer device are supported with
the help of virtual devices, which are created for each consumer device
- genpd pair. These are the device structures which are attached to the
power domain and are required by the OPP core to set the performance
state of the genpd.

The helpers added by this commit are required to be called once for each
of these virtual devices. These are required only if multiple domains
are available for a device, otherwise the actual device structure will
be used instead by the OPP core.

The new helpers also support the complex cases where the consumer device
wouldn't always require all the domains. For example, a camera may
require only one power domain during normal operations but two during
high resolution operations. The consumer driver can call
dev_pm_opp_put_genpd_virt_dev(high_resolution_genpd_virt_dev) if it is
currently operating in the normal mode and doesn't have any performance
requirements from the genpd which manages high resolution power
requirements. The consumer driver can later call
dev_pm_opp_set_genpd_virt_dev(high_resolution_genpd_virt_dev) once it
switches back to the high resolution mode.

The new helpers differ from other OPP set/put helpers as the new ones
can be called with OPPs initialized for the table as we may need to call
them on the fly because of the complex case explained above. For this
reason it is possible that the genpd virt_dev structure may be used in
parallel while the new helpers are running and a new mutex is added to
protect against that. We didn't use the existing opp_table->lock mutex
as that is widely used in the OPP core and we will need this lock in the
dev_pm_opp_set_rate() helper while changing OPP and we need to make sure
there is not much contention while doing that as that's the hotpath.
Reviewed-by: NUlf Hansson <ulf.hansson@linaro.org>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

An earlier commit populated the OPP tables from the "required-opps"
property, this commit populates the individual OPPs. This is repeated
for each OPP in the OPP table and these populated OPPs will be used by
later commits.
Reviewed-by: NUlf Hansson <ulf.hansson@linaro.org>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

The current implementation works only for the case where a single
phandle is present in the "required-opps" property, while DT allows
multiple phandles to be present there.

This patch adds new infrastructure to parse all the phandles present in
"required-opps" property and save pointers of the required OPP's OPP
tables. These will be used by later commits.
Reviewed-by: NUlf Hansson <ulf.hansson@linaro.org>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

Create a separate routine to take care of custom set_opp() handler
specific stuff.
Reviewed-by: NUlf Hansson <ulf.hansson@linaro.org>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

Return error number instead of 0 on failures.

Fixes: a1e8c136 ("PM / OPP: "opp-hz" is optional for power domains")
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

When two or more devices are sharing their clock and voltage rails, they
share the same OPP table. But there are some corner cases where the OPP
core incorrectly creates separate OPP tables for them.

For example, CPU 0 and 1 share clock/voltage rails. The platform
specific code calls dev_pm_opp_set_regulators() for CPU0 and the OPP
core creates an OPP table for it (the individual OPPs aren't initialized
as of now). The same is repeated for CPU1 then. Because
_opp_get_opp_table() doesn't compare DT node pointers currently, it
fails to find the link between CPU0 and CPU1 and so creates a new OPP
table.

Fix this by calling _managed_opp() from _opp_get_opp_table().
_managed_opp() gain an additional argument (index) to get the right node
pointer. This resulted in simplifying code in _of_add_opp_table_v2() as
well.
Tested-by: NNiklas Cassel <niklas.cassel@linaro.org>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

Currently there are two separate ways to free the OPP table based on how
it is created in the first place.

We call _dev_pm_opp_remove_table() to free the static and/or dynamic
OPP, OPP list devices, etc. This is done for the case where the OPP
table is added while initializing the OPPs, like via the path
dev_pm_opp_of_add_table().

We also call dev_pm_opp_put_opp_table() in some cases which eventually
frees the OPP table structure once the reference count reaches 0. This
is used by the first case as well as other cases like
dev_pm_opp_set_regulators() where the OPPs aren't necessarily
initialized at this point.

This whole thing is a bit unclear and messy and obstruct any further
cleanup/fixup of OPP core.

This patch tries to streamline this by keeping a single path for OPP
table destruction, i.e. dev_pm_opp_put_opp_table().

All the cleanup happens in _opp_table_kref_release() now after the
reference count reaches 0. _dev_pm_opp_remove_table() is removed as it
isn't required anymore.

We don't drop the reference to the OPP table after creating it from
_of_add_opp_table_v{1|2}() anymore and the same is dropped only when we
try to remove them.
Tested-by: NNiklas Cassel <niklas.cassel@linaro.org>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

Only one platform was depending on this feature and it is already
updated now. Stop removing dynamic OPPs from _dev_pm_opp_remove_table().
This simplifies lot of paths and removes unnecessary parameters.
Tested-by: NNiklas Cassel <niklas.cassel@linaro.org>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

The static OPPs don't always get freed with the OPP table, it can happen
before that as well. For example, if the OPP table is first created
using helpers like dev_pm_opp_set_supported_hw() and the OPPs are
created at a later point. Now when the OPPs are removed, the OPP table
stays until the time dev_pm_opp_put_supported_hw() is called.

Later patches will streamline the freeing of OPP table and that requires
the static OPPs to get freed with help of a separate kernel reference.
This patch prepares for that by creating a separate kref for static OPPs
list.
Tested-by: NNiklas Cassel <niklas.cassel@linaro.org>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

The reference count is only required to be incremented for every call
that may lead to adding the OPP table. For static OPPs the same should
be done from the parent routine which adds all static OPPs together and
so only one refcount for all static OPPs.

Update code to reflect that.

The refcount is incremented every time a dynamic OPP is created (as that
can lead to creating the OPP table) and the same is dropped when the OPP
is removed.
Tested-by: NNiklas Cassel <niklas.cassel@linaro.org>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

This is a preparatory patch required for the next commit which will
start using OPP table's node pointer in _of_init_opp_table(), which
requires the index in order to read the OPP table's phandle.

This commit adds the index argument in the call chains in order to get
it delivered to _of_init_opp_table().
Tested-by: NNiklas Cassel <niklas.cassel@linaro.org>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

The dev_list needs to be protected with a lock, else we may have
simultaneous access (addition/removal) to it and that would be racy.
Extend scope of the opp_table lock to protect dev_list as well.
Tested-by: NNiklas Cassel <niklas.cassel@linaro.org>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

This commit fixes a rare but possible case when the clk rate is updated
without update of the regulator voltage.

At boot up, CPUfreq checks if the system is running at the right freq. This
is a sanity check in case a bootloader set clk rate that is outside of freq
table present with cpufreq core. In such cases system can be unstable so
better to change it to a freq that is preset in freq-table.

The CPUfreq takes next freq that is >= policy->cur and this is our
target_freq that needs to be set now.

dev_pm_opp_set_rate(dev, target_freq) checks the target_freq and the
old_freq (a current rate). If these are equal it returns early. If not,
it searches for OPP (old_opp) that fits best to old_freq (not listed in
the table) and updates old_freq (!).

Here, we can end up with old_freq = old_opp.rate = target_freq, which
is not handled in _generic_set_opp_regulator(). It's supposed to update
voltage only when freq > old_freq  || freq > old_freq.

if (freq > old_freq) {
		ret = _set_opp_voltage(dev, reg, new_supply);
[...]
if (freq < old_freq) {
		ret = _set_opp_voltage(dev, reg, new_supply);
		if (ret)

It results in, no voltage update while clk rate is updated.

Example:
freq-table = {
	1000MHz   1.15V
	 666MHZ   1.10V
	 333MHz   1.05V
}
boot-up-freq        = 800MHz   # not listed in freq-table
freq = target_freq  = 1GHz
old_freq            = 800Mhz
old_opp = _find_freq_ceil(opp_table, &old_freq);  #(old_freq is modified!)
old_freq            = 1GHz

Fixes: 6a0712f6 ("PM / OPP: Add dev_pm_opp_set_rate()")
Cc: 4.6+ <stable@vger.kernel.org> # v4.6+
Signed-off-by: NWaldemar Rymarkiewicz <waldemar.rymarkiewicz@gmail.com>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

It should be fine to call dev_pm_opp_register_set_opp_helper() for all
possible CPUs, even if some of them share the OPP table as the caller
may not be aware of sharing policy.

Lets increment the reference count of the OPP table and return its
pointer. The caller need to call dev_pm_opp_register_put_opp_helper()
the same number of times later on to drop all the references.

To avoid adding another counter to count how many times
dev_pm_opp_register_set_opp_helper() is called for the same OPP table,
dev_pm_opp_register_put_opp_helper() frees the resources on the very
first call made to it, assuming that the caller would be calling it
sequentially for all the CPUs. We can revisit that if that assumption is
broken in the future.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

It should be fine to call dev_pm_opp_set_regulators() for all possible
CPUs, even if some of them share the OPP table as the caller may not be
aware of sharing policy.

Lets increment the reference count of the OPP table and return its
pointer. The caller need to call dev_pm_opp_put_regulators() the same
number of times later on to drop all the references.

To avoid adding another counter to count how many times
dev_pm_opp_set_regulators() is called for the same OPP table,
dev_pm_opp_put_regulators() frees the resources on the very first call
made to it, assuming that the caller would be calling it sequentially
for all the CPUs. We can revisit that if that assumption is broken in
the future.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

It should be fine to call dev_pm_opp_set_prop_name() for all possible
CPUs, even if some of them share the OPP table as the caller may not be
aware of sharing policy.

Lets increment the reference count of the OPP table and return its
pointer. The caller need to call dev_pm_opp_put_prop_name() the same
number of times later on to drop all the references.

To avoid adding another counter to count how many times
dev_pm_opp_set_prop_name() is called for the same OPP table,
dev_pm_opp_put_prop_name() frees the resources on the very first call
made to it, assuming that the caller would be calling it sequentially
for all the CPUs. We can revisit that if that assumption is broken in
the future.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

It should be fine to call dev_pm_opp_set_supported_hw() for all possible
CPUs, even if some of them share the OPP table as the caller may not be
aware of sharing policy.

Lets increment the reference count of the OPP table and return its
pointer. The caller need to call dev_pm_opp_put_supported_hw() the same
number of times later on to drop all the references.

To avoid adding another counter to count how many times
dev_pm_opp_set_supported_hw() is called for the same OPP table,
dev_pm_opp_put_supported_hw() frees the resources on the very first call
made to it, assuming that the caller would be calling it sequentially
for all the CPUs. We can revisit that if that assumption is broken in
the future.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

These helpers aren't used anymore, remove them.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Reviewed-by: NUlf Hansson <ulf.hansson@linaro.org>

The genpd core provides an API now to retrieve the performance state
from DT, use that instead of the ->get_pstate() callback.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Reviewed-by: NUlf Hansson <ulf.hansson@linaro.org>

A device's DT node or its OPP nodes can contain a phandle to other
device's OPP node, in the "required-opps" property.

This patch implements a routine to find that required OPP from the node
that contains the "required-opps" property.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Reviewed-by: NUlf Hansson <ulf.hansson@linaro.org>

"opp-hz" property is optional for power domains now and we shouldn't
error out if it is missing for power domains.

This patch creates two new routines, _get_opp_count() and
_opp_is_duplicate(), by separating existing code from their parent
functions. Also skip duplicate OPP check for power domain OPPs as they
may not have any the "opp-hz" field, but a platform specific performance
state binding to uniquely identify OPP nodes.

By default the debugfs OPP nodes are named using the "rate" value, but
that isn't possible for the power domain OPP nodes and hence they use
the index of the OPP node in the OPP node list instead.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Reviewed-by: NUlf Hansson <ulf.hansson@linaro.org>

This adds the dev_pm_opp_{un}register_get_pstate_helper() helper
routines which will be used to set the get_pstate() callback for a
device. This callback will be later called internally by the OPP core to
get performance state corresponding to an OPP.

This is required temporarily until the time we have proper DT bindings
to include the performance state information.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The genpd framework now provides an API to request device's power
domain to update its performance state.  Use that interface from the
OPP core for devices whose power domains support performance states.

Note that this commit doesn't add any mechanism by which performance
states are made available to the OPP core. That would be done by a
later commit.

Note that the current implementation is restricted to the case where
the device doesn't have separate regulators for itself. We shouldn't
over engineer the code before we have real use case for them. We can
always come back and add more code to support such cases later on.
Tested-by: NRajendra Nayak <rnayak@codeaurora.org>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The routine is named incorrectly since the first attempt as there is
nothing like a put_opp() helper. We wanted to unregister the set_opp()
helper here and so it should rather be named as
dev_pm_opp_unregister_set_opp_helper().
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Reviewed-by: NStephen Boyd <sboyd@codeaurora.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

On some i.MX6 platforms which do not have speed grading
check, opp table will not be created in platform code,
so cpufreq driver prints the following error message:

cpu cpu0: dev_pm_opp_get_opp_count: OPP table not found (-19)

However, this is not really an error in this case because the
imx6q-cpufreq driver first calls dev_pm_opp_get_opp_count()
and if it fails, it means that platform code does not provide
OPP and then dev_pm_opp_of_add_table() will be called.

In order to avoid such confusing error message, move it to
debug level.

It is up to the caller of dev_pm_opp_get_opp_count() to check its
return value and decide if it will print an error or not.
Signed-off-by: NFabio Estevam <fabio.estevam@nxp.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The drivers/base/power/ directory is special and contains code related
to power management core like system suspend/resume, hibernation, etc.
It was fine to keep the OPP code inside it when we had just one file for
it, but it is growing now and already has a directory for itself.

Lets move it directly under drivers/ directory, just like cpufreq and
cpuidle.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Acked-by: NStephen Boyd <sboyd@codeaurora.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The notifier callbacks may want to call some OPP helper routines which
may try to take the same opp_table->lock again and cause a deadlock. One
such usecase was reported by Chanwoo Choi, where calling
dev_pm_opp_disable() leads us to the devfreq's OPP notifier handler,
which further calls dev_pm_opp_find_freq_floor() and it deadlocks.

We don't really need the opp_table->lock to be held across the notifier
call though, all we want to make sure is that the 'opp' doesn't get
freed while being used from within the notifier chain. We can do it with
help of dev_pm_opp_get/put() as well. Let's do it.

Cc: 4.11+ <stable@vger.kernel.org> # 4.11+
Fixes: 5b650b38 "PM / OPP: Take kref from _find_opp_table()"
Reported-by: NChanwoo Choi <cw00.choi@samsung.com>
Tested-by: NChanwoo Choi <cw00.choi@samsung.com>
Reviewed-by: NStephen Boyd <sboyd@codeaurora.org>
Reviewed-by: NChanwoo Choi <cw00.choi@samsung.com>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

In order to support OPP switching, OPP layer needs to get pointer to the
clock for the device. Simple cases work fine without using the routines
added by this patch (i.e.  by passing connection-id as NULL), but for a
device with multiple clocks available, the OPP core needs to know the
exact name of the clk to use.

Add a new set of APIs to get that done.
Tested-by: NRajendra Nayak <rnayak@codeaurora.org>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Reviewed-by: NStephen Boyd <sboyd@codeaurora.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

This code was required while the OPP core was managed with help of RCUs,
but not anymore. Get rid of unnecessary alloc/memcpy operations.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Reviewed-by: NStephen Boyd <sboyd@codeaurora.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The code was overly complicated here because of the limitations that we
had with RCUs (Couldn't use opp-table and OPPs outside RCU protected
section and can't call sleep-able routines from within that). But that
is long gone now.

Reorganize _generic_set_opp_regulator() in order to avoid using "struct
dev_pm_set_opp_data" and copying data into it for the case where
opp_table->set_opp is not set.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Reviewed-by: NStephen Boyd <sboyd@codeaurora.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Reading array at given index before checking if index is valid results in
illegal memory access.

The bug was detected using KASAN framework.
Signed-off-by: NAndrzej Hajda <a.hajda@samsung.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Fixes the following sparse warning:

drivers/base/power/opp/core.c:49:18: warning:
 symbol '_find_opp_table_unlocked' was not declared. Should it be static?
Signed-off-by: NWei Yongjun <weiyongjun1@huawei.com>
Reviewed-by: NStephen Boyd <sboyd@codeaurora.org>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

dev_pm_opp_get_max_volt_latency() calls _find_opp_table() two times
effectively.

Merge _get_regulator_count() into dev_pm_opp_get_max_volt_latency() to
avoid that.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Reviewed-by: NStephen Boyd <sboyd@codeaurora.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

As we don't use RCU locking anymore, there is no need to replace an
earlier OPP node with a new one. Just update the existing one.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Reviewed-by: NStephen Boyd <sboyd@codeaurora.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The RCU locking isn't well suited for the OPP core. The RCU locking fits
better for reader heavy stuff, while the OPP core have at max one or two
readers only at a time.

Over that, it was getting very confusing the way RCU locking was used
with the OPP core. The individual OPPs are mostly well handled, i.e. for
an update a new structure was created and then that replaced the older
one. But the OPP tables were updated directly all the time from various
parts of the core. Though they were mostly used from within RCU locked
region, they didn't had much to do with RCU and were governed by the
mutex instead.

And that mixed with the 'opp_table_lock' has made the core even more
confusing.

Now that we are already managing the OPPs and the OPP tables with kernel
reference infrastructure, we can get rid of RCU locking completely and
simplify the code a lot.

Remove all RCU references from code and comments.

Acquire opp_table->lock while parsing the list of OPPs though.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Reviewed-by: NStephen Boyd <sboyd@codeaurora.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Take reference of the OPP table from within _find_opp_table(). Also
update the callers of _find_opp_table() to call
dev_pm_opp_put_opp_table() after they have used the OPP table.

Note that _find_opp_table() increments the reference under the
opp_table_lock.

Now that the OPP table wouldn't get freed until the callers of
_find_opp_table() call dev_pm_opp_put_opp_table(), there is no need to
take the opp_table_lock or rcu_read_lock() around it. Drop them.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Reviewed-by: NStephen Boyd <sboyd@codeaurora.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

This patch updates dev_pm_opp_find_freq_*() routines to get a reference
to the OPPs returned by them.

Also updates the users of dev_pm_opp_find_freq_*() routines to call
dev_pm_opp_put() after they are done using the OPPs.

As it is guaranteed the that OPPs wouldn't get freed while being used,
the RCU read side locking present with the users isn't required anymore.
Drop it as well.

This patch also updates all users of devfreq_recommended_opp() which was
returning an OPP received from the OPP core.

Note that some of the OPP core routines have gained
rcu_read_{lock|unlock}() calls, as those still use RCU specific APIs
within them.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Reviewed-by: Chanwoo Choi <cw00.choi@samsung.com> [Devfreq]
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Add kref to struct dev_pm_opp for easier accounting of the OPPs.

Note that the OPPs are freed under the opp_table->lock mutex only.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Reviewed-by: NStephen Boyd <sboyd@codeaurora.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Migrate all users of _add_opp_table() to use dev_pm_opp_get_opp_table()
to guarantee that the OPP table doesn't get freed while being used.

Also update _managed_opp() to get the reference to the OPP table.

Now that the OPP table wouldn't get freed while these routines are
executing after dev_pm_opp_get_opp_table() is called, there is no need
to take opp_table_lock. Drop them as well.

Now that _add_opp_table(), _remove_opp_table() and the unlocked release
routines aren't used anymore, remove them.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Reviewed-by: NStephen Boyd <sboyd@codeaurora.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>