Enable HWP boost on Skylake server and workstations.
Reported-by: NMel Gorman <mgorman@techsingularity.net>
Tested-by: NGiovanni Gherdovich <ggherdovich@suse.cz>
Signed-off-by: NSrinivas Pandruvada <srinivas.pandruvada@linux.intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

A new attribute is added to intel_pstate sysfs to enable/disable
HWP dynamic performance boost.
Reported-by: NMel Gorman <mgorman@techsingularity.net>
Tested-by: NGiovanni Gherdovich <ggherdovich@suse.cz>
Signed-off-by: NSrinivas Pandruvada <srinivas.pandruvada@linux.intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

This change uses SCHED_CPUFREQ_IOWAIT flag to boost HWP performance.
Since SCHED_CPUFREQ_IOWAIT flag is set frequently, we don't start
boosting steps unless we see two consecutive flags in two ticks. This
avoids boosting due to IO because of regular system activities.

To avoid synchronization issues, the actual processing of the flag is
done on the local CPU callback.
Reported-by: NMel Gorman <mgorman@techsingularity.net>
Tested-by: NGiovanni Gherdovich <ggherdovich@suse.cz>
Signed-off-by: NSrinivas Pandruvada <srinivas.pandruvada@linux.intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Added two utility functions to HWP boost up gradually and boost down to
the default cached HWP request values.

Boost up:
Boost up updates HWP request minimum value in steps. This minimum value
can reach upto at HWP request maximum values depends on how frequently,
this boost up function is called. At max, boost up will take three steps
to reach the maximum, depending on the current HWP request levels and HWP
capabilities. For example, if the current settings are:
If P0 (Turbo max) = P1 (Guaranteed max) = min
        No boost at all.
If P0 (Turbo max) > P1 (Guaranteed max) = min
        Should result in one level boost only for P0.
If P0 (Turbo max) = P1 (Guaranteed max) > min
        Should result in two level boost:
                (min + p1)/2 and P1.
If P0 (Turbo max) > P1 (Guaranteed max) > min
        Should result in three level boost:
                (min + p1)/2, P1 and P0.
We don't set any level between P0 and P1 as there is no guarantee that
they will be honored.

Boost down:
After the system is idle for hold time of 3ms, the HWP request is reset
to the default value from HWP init or user modified one via sysfs.

Caching of HWP Request and Capabilities
Store the HWP request value last set using MSR_HWP_REQUEST and read
MSR_HWP_CAPABILITIES. This avoid reading of MSRs in the boost utility
functions.

These boost utility functions calculated limits are based on the latest
HWP request value, which can be modified by setpolicy() callback. So if
user space modifies the minimum perf value, that will be accounted for
every time the boost up is called. There will be case when there can be
contention with the user modified minimum perf, in that case user value
will gain precedence. For example just before HWP_REQUEST MSR is updated
from setpolicy() callback, the boost up function is called via scheduler
tick callback. Here the cached MSR value is already the latest and limits
are updated based on the latest user limits, but on return the MSR write
callback called from setpolicy() callback will update the HWP_REQUEST
value. This will be used till next time the boost up function is called.

In addition add a variable to control HWP dynamic boosting. When HWP
dynamic boost is active then set the HWP specific update util hook. The
contents in the utility hooks will be filled in the subsequent patches.
Reported-by: NMel Gorman <mgorman@techsingularity.net>
Tested-by: NGiovanni Gherdovich <ggherdovich@suse.cz>
Signed-off-by: NSrinivas Pandruvada <srinivas.pandruvada@linux.intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The ti_cpufreq_probe() function uses regular kzalloc to allocate
the ti_cpufreq_data structure and kfree for freeing this memory
on failures. Simplify this code by using the devres managed
API.
Signed-off-by: NSuman Anna <s-anna@ti.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Commit 05829d94 (cpufreq: ti-cpufreq: kfree opp_data when
failure) has fixed a memory leak in the failure path, however
the patch returned a positive value on get_cpu_device() failure
instead of the previous negative value. Fix this incorrect error
return value properly.

Fixes: 05829d94 (cpufreq: ti-cpufreq: kfree opp_data when failure)
Cc: 4.14+ <stable@vger.kernel.org> # v4.14+
Signed-off-by: NSuman Anna <s-anna@ti.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The acpi_cpufreq_fast_switch() function is local to the source and
does not need to be in global scope, so make it static.

Cleans up sparse warning:
drivers/cpufreq/acpi-cpufreq.c:468:14: warning: symbol
'acpi_cpufreq_fast_switch' was not declared. Should it be static?
Signed-off-by: NColin Ian King <colin.king@canonical.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Building the kryo cpufreq driver while QCOM_SMEM is a loadable module
results in a link error:

drivers/cpufreq/qcom-cpufreq-kryo.o: In function `qcom_cpufreq_kryo_probe':
qcom-cpufreq-kryo.c:(.text+0xbc): undefined reference to `qcom_smem_get'

The problem is that Kconfig ignores interprets the dependency as met
when the dependent symbol is a 'bool' one. By making it 'tristate',
it will be forced to be a module here, which builds successfully.

Fixes: 46e2856b (cpufreq: Add Kryo CPU scaling driver)
Signed-off-by: NArnd Bergmann <arnd@arndb.de>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

In Certain QCOM SoCs like apq8096 and msm8996 that have KRYO processors,
the CPU frequency subset and voltage value of each OPP varies
based on the silicon variant in use. Qualcomm Process Voltage Scaling Tables
defines the voltage and frequency value based on the msm-id in SMEM
and speedbin blown in the efuse combination.
The qcom-cpufreq-kryo driver reads the msm-id and efuse value from the SoC
to provide the OPP framework with required information.
This is used to determine the voltage and frequency value for each OPP of
operating-points-v2 table when it is parsed by the OPP framework.
Signed-off-by: NIlia Lin <ilialin@codeaurora.org>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Reviewed-by: NAmit Kucheria <amit.kucheria@linaro.org>
Tested-by: NAmit Kucheria <amit.kucheria@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Use the static SRCU initializer for `cpufreq_transition_notifier_list'.
This avoids the init_cpufreq_transition_notifier_list() initcall. Its
only purpose is to initialize the SRCU notifier once during boot and set
another variable which is used as an indicator whether the init was
perfromed before cpufreq_register_notifier() was used.
Signed-off-by: NSebastian Andrzej Siewior <bigeasy@linutronix.de>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

If the policy limits are updated via cpufreq_update_policy() and
subsequently via sysfs, the limits stored in user_policy may be
set incorrectly.

For example, if both min and max are set via sysfs to the maximum
available frequency, user_policy.min and user_policy.max will also
be the maximum.  If a policy notifier triggered by
cpufreq_update_policy() lowers both the min and the max at this
point, that change is not reflected by the user_policy limits, so
if the max is updated again via sysfs to the same lower value,
then user_policy.max will be lower than user_policy.min which
shouldn't happen.  In particular, if one of the policy CPUs is
then taken offline and back online, cpufreq_set_policy() will
fail for it due to a failing limits check.

To prevent that from happening, initialize the min and max fields
of the new_policy object to the ones stored in user_policy that
were previously set via sysfs.
Signed-off-by: NKevin Wangtao <kevin.wangtao@hisilicon.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
[ rjw: Subject & changelog ]
Cc: All applicable <stable@vger.kernel.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

This routine checks if the CPU running this code belongs to the policy
of the target CPU or if not, can it do remote DVFS for it remotely. But
the current name of it implies as if it is only about doing remote
updates.

Rename it to make it more relevant.
Suggested-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Currently tegra20-cpufreq kernel module isn't getting autoloaded because
there is no device associated with the module, this is one of two patches
that resolves the module autoloading. This patch adds a module alias that
will associate the tegra20-cpufreq kernel module with the platform device,
other patch will instantiate the actual platform device. And now it makes
sense to wrap cpufreq driver into a platform driver for consistency.
Signed-off-by: NDmitry Osipenko <digetx@gmail.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Nothing prevents Tegra20 CPUFreq module to be unloaded, hence allow it to
be built as a non-builtin kernel module.
Signed-off-by: NDmitry Osipenko <digetx@gmail.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Acked-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Don't even try to request the clocks during of module initialization on
non-Tegra20 machines (this is the case for a multi-platform kernel) for
consistency.
Signed-off-by: NDmitry Osipenko <digetx@gmail.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Acked-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Remove unneeded variable initialization solely for consistency.
Signed-off-by: NDmitry Osipenko <digetx@gmail.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Acked-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Remove unnecessary parentheses as suggested by the checkpatch script.
Signed-off-by: NDmitry Osipenko <digetx@gmail.com>
Acked-by: NThierry Reding <treding@nvidia.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Remove checking of the CPU number for consistency as it won't ever fail
unless there is a severe bug in the cpufreq core.
Signed-off-by: NDmitry Osipenko <digetx@gmail.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Acked-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Properly put requested clocks in the module init/exit code.
Signed-off-by: NDmitry Osipenko <digetx@gmail.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Acked-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The EMC driver has been gone 4 years ago, since the commit a7cbe92c
("ARM: tegra: remove tegra EMC scaling driver"). Remove the EMC clock
usage as it does nothing. We may consider re-implementing the EMC scaling
later, probably using PM Memory Bandwidth QoS API.
Signed-off-by: NDmitry Osipenko <digetx@gmail.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Acked-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Remove unused/unneeded headers and sort them in the alphabet order.
Signed-off-by: NDmitry Osipenko <digetx@gmail.com>
Acked-by: NThierry Reding <treding@nvidia.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Remove unneeded blank line and replace whitespaces with a tab in the code
for consistency.
Signed-off-by: NDmitry Osipenko <digetx@gmail.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Acked-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Change module description to be in line with the other Tegra drivers, just
for consistency.
Signed-off-by: NDmitry Osipenko <digetx@gmail.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Acked-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

This reverts commit 034def59.

This is no longer needed since the following commit and to the best
of my knowledge is not relied on by any upstream DTS:

edeec420 (cpufreq: dt-platdev: Automatically create cpufreq
device with OPP v2)
Signed-off-by: NSimon Horman <horms+renesas@verge.net.au>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

This reverts commit bea2ebca.

This is no longer needed since the following commit and to the best
of my knowledge is not relied on by any upstream DTS:

edeec420 (cpufreq: dt-platdev: Automatically create cpufreq
device with OPP v2)
Signed-off-by: NSimon Horman <horms+renesas@verge.net.au>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Allow use of the trace_pstate_sample trace function
when the intel_pstate driver is in passive mode.
Since the core_busy and scaled_busy fields are not
used, and it might be desirable to know which path
through the driver was used, either intel_cpufreq_target
or intel_cpufreq_fast_switch, re-task the core_busy
field as a flag indicator.

The user can then use the intel_pstate_tracer.py utility
to summarize and plot the trace.

Note: The core_busy feild still goes by that name
in include/trace/events/power.h and within the
intel_pstate_tracer.py script and csv file headers,
but it is graphed as "performance", and called
core_avg_perf now in the intel_pstate driver.

Sometimes, in passive mode, the driver is not called for
many tens or even hundreds of seconds. The user
needs to understand, and not be confused by, this limitation.
Signed-off-by: NDoug Smythies <dsmythies@telus.net>
Signed-off-by: NSrinivas Pandruvada <srinivas.pandruvada@linux.intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Armada-37xx driver registers a cpufreq-dt driver. Not having
CONFIG_CPUFREQ_DT selected leads to a silent abort during the probe.
Prevent that situation by having the former depending on the latter.

Fixes: 92ce45fb (cpufreq: Add DVFS support for Armada 37xx)
Cc: 4.16+ <stable@vger.kernel.org> # 4.16+
Signed-off-by: NMiquel Raynal <miquel.raynal@bootlin.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

cpufreq_notify_transition() calls __cpufreq_notify_transition() for each
CPU of a policy. There is a lot of code in __cpufreq_notify_transition()
though which isn't required to be executed for each CPU, like checking
about disabled cpufreq or irqs, adjusting jiffies, updating cpufreq
stats and some debug print messages.

This commit merges __cpufreq_notify_transition() into
cpufreq_notify_transition() and modifies cpufreq_notify_transition() to
execute minimum amount of code for each CPU.

Also fix the kerneldoc for cpufreq_notify_transition() while at it.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Trivial fix to spelling mistake in s3c_freq_dbg debug message text.
Signed-off-by: NColin Ian King <colin.king@canonical.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

speedstep_detect_processor() is declared as returing an
'enum speedstep_processor' but use an 'int' in its definition.

Fix this by using 'enum speedstep_processor' in its definition too.
Signed-off-by: NLuc Van Oostenryck <luc.vanoostenryck@gmail.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Add suspend/resume hooks in Armada 37xx DVFS driver to handle S2RAM
operations. As there is currently no 'driver' structure, create one
to store both the regmap and the register values during suspend
operation.
Signed-off-by: NMiquel Raynal <miquel.raynal@bootlin.com>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Tested-by: NGregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The resources weren't freed on failures, free them properly.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Tested-by: NMiquel Raynal <miquel.raynal@bootlin.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Platforms may need to implement platform specific suspend/resume hooks.
Update cpufreq-dt driver's platform data to contain those for such
platforms.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Tested-by: NMiquel Raynal <miquel.raynal@bootlin.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Add support to specify platform specific transition_delay_us instead
of using the transition delay derived from PCC.

With commit 3d41386d (cpufreq: CPPC: Use transition_delay_us
depending transition_latency) we are setting transition_delay_us
directly and not applying the LATENCY_MULTIPLIER. Because of that,
on Qualcomm Centriq we can end up with a very high rate of frequency
change requests when using the schedutil governor (default
rate_limit_us=10 compared to an earlier value of 10000).

The PCC subspace describes the rate at which the platform can accept
commands on the CPPC's PCC channel. This includes read and write
command on the PCC channel that can be used for reasons other than
frequency transitions. Moreover the same PCC subspace can be used by
multiple freq domains and deriving transition_delay_us from it as we
do now can be sub-optimal.

Moreover if a platform does not use PCC for desired_perf register then
there is no way to compute the transition latency or the delay_us.

CPPC does not have a standard defined mechanism to get the transition
rate or the latency at the moment.

Given the above limitations, it is simpler to have a platform specific
transition_delay_us and rely on PCC derived value only if a platform
specific value is not available.
Signed-off-by: NPrashanth Prakash <pprakash@codeaurora.org>
Cc: 4.14+ <stable@vger.kernel.org> # 4.14+
Fixes: 3d41386d (cpufreq: CPPC: Use transition_delay_us depending transition_latency)
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

gpstate_timer_handler() uses synchronous smp_call to set the pstate
on the requested core. This causes the below hard lockup:

  smp_call_function_single+0x110/0x180 (unreliable)
  smp_call_function_any+0x180/0x250
  gpstate_timer_handler+0x1e8/0x580
  call_timer_fn+0x50/0x1c0
  expire_timers+0x138/0x1f0
  run_timer_softirq+0x1e8/0x270
  __do_softirq+0x158/0x3e4
  irq_exit+0xe8/0x120
  timer_interrupt+0x9c/0xe0
  decrementer_common+0x114/0x120
  -- interrupt: 901 at doorbell_global_ipi+0x34/0x50
  LR = arch_send_call_function_ipi_mask+0x120/0x130
  arch_send_call_function_ipi_mask+0x4c/0x130
  smp_call_function_many+0x340/0x450
  pmdp_invalidate+0x98/0xe0
  change_huge_pmd+0xe0/0x270
  change_protection_range+0xb88/0xe40
  mprotect_fixup+0x140/0x340
  SyS_mprotect+0x1b4/0x350
  system_call+0x58/0x6c

One way to avoid this is removing the smp-call. We can ensure that the
timer always runs on one of the policy-cpus. If the timer gets
migrated to a cpu outside the policy then re-queue it back on the
policy->cpus. This way we can get rid of the smp-call which was being
used to set the pstate on the policy->cpus.

Fixes: 7bc54b65 ("timers, cpufreq/powernv: Initialize the gpstate timer as pinned")
Cc: stable@vger.kernel.org # v4.8+
Reported-by: NNicholas Piggin <npiggin@gmail.com>
Reported-by: NPridhiviraj Paidipeddi <ppaidipe@linux.vnet.ibm.com>
Signed-off-by: NShilpasri G Bhat <shilpa.bhat@linux.vnet.ibm.com>
Acked-by: NNicholas Piggin <npiggin@gmail.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Acked-by: NVaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
Signed-off-by: NMichael Ellerman <mpe@ellerman.id.au>

This debug code was helpful while developing the driver, but it isn't
being used for anything anymore.
Signed-off-by: NMarkus Mayer <mmayer@broadcom.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

This isn't used anymore. Remove the helper and update documentation
accordingly.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The cpufreq core is already validating the CPU frequency table after
calling the ->init() callback of the cpufreq drivers and the drivers
don't need to do the same anymore. Though they need to set the
policy->freq_table field directly from the ->init() callback now.

Stop validating the frequency table from SCMI driver.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Acked-by: NSudeep Holla <sudeep.holla@arm.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

When multiple CPUs are related in one cpufreq policy, the first online
CPU will be chosen by default to handle cpufreq operations. Let's take
cpu0 and cpu1 as an example.

When cpu0 is offline, policy->cpu will be shifted to cpu1. cpu1's perf
capabilities should be initialized. Otherwise, perf capabilities are 0s
and speed change can not take effect.

This patch copies perf capabilities of the first online CPU to other
shared CPUs when policy shared type is CPUFREQ_SHARED_TYPE_ANY.
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NShunyong Yang <shunyong.yang@hxt-semitech.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

There was no clk_put() balancing the clk_get(). This commit fixes it.

Fixes: 92ce45fb (cpufreq: Add DVFS support for Armada 37xx)
Cc: 4.16+ <stable@vger.kernel.org> # 4.16+
Reported-by: NThomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: NGregory CLEMENT <gregory.clement@bootlin.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>