It is possible that some performance excursions happened before OS boot
or enable HWP interrupts. So clear MSR_HWP_STATUS bits when we enable
HWP interrupt. In this way a next excursion will results in a HWP
interrupt.

The status bits of MSR_HWP_STATUS must be cleared (0) by software so
that a new status condition change will cause the hardware to set the
bit again and issue the notification.

Fixes: 57577c99 ("cpufreq: intel_pstate: Process HWP Guaranteed change notification")
Signed-off-by: NSrinivas Pandruvada <srinivas.pandruvada@linux.intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

It is possible that on some platforms HWP interrupts are disabled. In
that case accessing MSR 0x773 will result in warning.

So check X86_FEATURE_HWP_NOTIFY feature to access MSR 0x773. The other
places in code where this MSR is accessed, already checks this feature
except during disable path called during cpufreq offline and suspend
callbacks.

Fixes: 57577c99 ("cpufreq: intel_pstate: Process HWP Guaranteed change notification")
Reported-by: NSteven Rostedt <rostedt@goodmis.org>
Signed-off-by: NSrinivas Pandruvada <srinivas.pandruvada@linux.intel.com>
Tested-by: NSteven Rostedt (VMware) <rostedt@goodmis.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Commit a365ab6b ("cpufreq: intel_pstate: Implement the
->adjust_perf() callback") caused intel_pstate to use nonzero HWP
desired values in certain usage scenarios, but it did not prevent
them from being leaked into the confugirations in which HWP desired
is expected to be 0.

The failing scenarios are switching the driver from the passive
mode to the active mode and starting a new kernel via kexec() while
intel_pstate is running in the passive mode.

To address this issue, ensure that HWP desired will be cleared on
offline and suspend/shutdown.

Fixes: a365ab6b ("cpufreq: intel_pstate: Implement the ->adjust_perf() callback")
Reported-by: NJulia Lawall <julia.lawall@inria.fr>
Tested-by: NJulia Lawall <julia.lawall@inria.fr>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Fix a problem in active mode that cpu->pstate.turbo_freq is initialized
only if HWP-to-frequency scaling factor is refined.

In passive mode, this problem is not exposed, because
cpu->pstate.turbo_freq is set again, later in
intel_cpufreq_cpu_init()->intel_pstate_get_hwp_cap().

Fixes: eb3693f0 ("cpufreq: intel_pstate: hybrid: CPU-specific scaling factor")
Signed-off-by: NZhang Rui <rui.zhang@intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The new cpufreq table flag RELATION_E introduced a new "efficient"
parameter for the cpufreq_table_find*() functions.

Fixes: 1f39fa0d (cpufreq: Introducing CPUFREQ_RELATION_E)
Signed-off-by: NVincent Donnefort <vincent.donnefort@arm.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Let the governors schedutil, conservative and ondemand to work, if possible
on efficient frequencies only.
Signed-off-by: NVincent Donnefort <vincent.donnefort@arm.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

This newly introduced flag can be applied by a governor to a CPUFreq
relation, when looking for a frequency within the policy table. The
resolution would then only walk through efficient frequencies.

Even with the flag set, the policy max limit will still be honoured. If no
efficient frequencies can be found within the limits of the policy, an
inefficient one would be returned.
Signed-off-by: NVincent Donnefort <vincent.donnefort@arm.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

When applying the policy min/max limits, the requested frequency is
simply clamped to not be out of range. It means, however, if one of the
boundaries isn't an available frequency, the frequency resolution can
return a value out of those limits, depending on the relation used.

e.g. freq{0,1,2} being available frequencies.

          freq0  policy->min  freq1  policy->max   freq2
            |        |          |        |           |
          17kHz     18kHz     19kHz     20kHz      21kHz

     __resolve_freq(21kHz, CPUFREQ_RELATION_L) -> 21kHz (out of bounds)
     __resolve_freq(17kHz, CPUFREQ_RELATION_H) -> 17kHz (out of bounds)

If, during the policy init, we resolve the requested min/max to existing
frequencies, we ensure that any CPUFREQ_RELATION_* would resolve to a
frequency which is inside the policy min/max range.

Making the policy limits rigid helps to introduce the inefficient
frequencies support. Resolving an inefficient frequency to an efficient
one should not transgress policy->max (which can be set for thermal
reason) and having a value we can trust simplify this comparison.
Signed-off-by: NVincent Donnefort <vincent.donnefort@arm.com>
Acked-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

It is possible that HWP guaranteed ratio is changed in response to
change in power and thermal limits. For example when Intel Speed Select
performance profile is changed or there is change in TDP, hardware can
send notifications. It is possible that the guaranteed ratio is
increased. This creates an issue when turbo is disabled, as the old
limits set in MSR_HWP_REQUEST are still lower and hardware will clip
to older limits.

This change enables HWP interrupt and process HWP interrupts. When
guaranteed is changed, calls cpufreq_update_policy() so that driver
callbacks are called to update to new HWP limits. This callback
is called from a delayed workqueue of 10ms to avoid frequent updates.

Although the scope of IA32_HWP_INTERRUPT is per logical cpu, on some
plaforms interrupt is generated on all CPUs. This is particularly a
problem during initialization, when the driver didn't allocated
data for other CPUs. So this change uses a cpumask of enabled CPUs and
process interrupts on those CPUs only.

When the cpufreq offline() or suspend() callback is called, HWP interrupt
is disabled on those CPUs and also cancels any pending work item.

Spin lock is used to protect data and processing shared with interrupt
handler. Here READ_ONCE(), WRITE_ONCE() macros are used to designate
shared data, even though spin lock act as an optimization barrier here.
Signed-off-by: NSrinivas Pandruvada <srinivas.pandruvada@linux.intel.com>
Tested-by: pablomh@gmail.com
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The return value from tegra_bpmp_transfer indicates the success or
failure of the IPC transaction with BPMP. If the transaction
succeeded, we also need to check the actual command's result code.
Add code to do this.

While at it, explicitly handle missing CPU clusters, which can
occur on floorswept chips. This worked before as well, but
possibly only by accident.
Signed-off-by: NMikko Perttunen <mperttunen@nvidia.com>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

list cpu_data_list has been inited staticly through LIST_HEAD,
so there's no need to call another INIT_LIST_HEAD. Simply remove
it from cppc_cpufreq_init.
Signed-off-by: NHan Wang <zjuwanghan@outlook.com>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

Apparently nobody has so far caught this warning, I hit it in randconfig
build testing:

drivers/cpufreq/s3c2440-cpufreq.c: In function 's3c2440_cpufreq_setdivs':
drivers/cpufreq/s3c2440-cpufreq.c:175:10: error: this statement may fall through [-Werror=implicit-fallthrough=]
   camdiv |= S3C2440_CAMDIVN_HCLK3_HALF;
          ^
drivers/cpufreq/s3c2440-cpufreq.c:176:2: note: here
  case 3:
  ^~~~
drivers/cpufreq/s3c2440-cpufreq.c:181:10: error: this statement may fall through [-Werror=implicit-fallthrough=]
   camdiv |= S3C2440_CAMDIVN_HCLK4_HALF;
          ^
drivers/cpufreq/s3c2440-cpufreq.c:182:2: note: here
  case 4:
  ^~~~

Both look like the fallthrough is intentional, so add the new
"fallthrough;" keyword.
Signed-off-by: NArnd Bergmann <arnd@arndb.de>
Acked-by: NKrzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

arm:allmodconfig fails to build with the following error.

drivers/cpufreq/vexpress-spc-cpufreq.c:454:13: error:
					unused variable 'cur_cluster'

Remove the unused variable.

Fixes: bb8c26d9 ("cpufreq: vexpress: Set CPUFREQ_IS_COOLING_DEV flag")
Cc: Viresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NGuenter Roeck <linux@roeck-us.net>
Reviewed-by: NKees Cook <keescook@chromium.org>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

arm:allmodconfig fails to build with the following error.

  drivers/cpufreq/vexpress-spc-cpufreq.c:454:13: error:
					unused variable 'cur_cluster'

Remove the unused variable.

Fixes: bb8c26d9 ("cpufreq: vexpress: Set CPUFREQ_IS_COOLING_DEV flag")
Cc: Viresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NGuenter Roeck <linux@roeck-us.net>
Reviewed-by: NKees Cook <keescook@chromium.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Since commit e5c6b312 ("cpufreq: schedutil: Use kobject release()
method to free sugov_tunables") kobject_put() has kfree()d the
attr_set before gov_attr_set_put() returns.

kobject_put() isn't the last user of attr_set in gov_attr_set_put(),
the subsequent mutex_destroy() triggers a use-after-free:
| BUG: KASAN: use-after-free in mutex_is_locked+0x20/0x60
| Read of size 8 at addr ffff000800ca4250 by task cpuhp/2/20
|
| CPU: 2 PID: 20 Comm: cpuhp/2 Not tainted 5.15.0-rc1 #12369
| Hardware name: ARM LTD ARM Juno Development Platform/ARM Juno Development
| Platform, BIOS EDK II Jul 30 2018
| Call trace:
|  dump_backtrace+0x0/0x380
|  show_stack+0x1c/0x30
|  dump_stack_lvl+0x8c/0xb8
|  print_address_description.constprop.0+0x74/0x2b8
|  kasan_report+0x1f4/0x210
|  kasan_check_range+0xfc/0x1a4
|  __kasan_check_read+0x38/0x60
|  mutex_is_locked+0x20/0x60
|  mutex_destroy+0x80/0x100
|  gov_attr_set_put+0xfc/0x150
|  sugov_exit+0x78/0x190
|  cpufreq_offline.isra.0+0x2c0/0x660
|  cpuhp_cpufreq_offline+0x14/0x24
|  cpuhp_invoke_callback+0x430/0x6d0
|  cpuhp_thread_fun+0x1b0/0x624
|  smpboot_thread_fn+0x5e0/0xa6c
|  kthread+0x3a0/0x450
|  ret_from_fork+0x10/0x20

Swap the order of the calls.

Fixes: e5c6b312 ("cpufreq: schedutil: Use kobject release() method to free sugov_tunables")
Cc: 4.7+ <stable@vger.kernel.org> # 4.7+
Signed-off-by: NJames Morse <james.morse@arm.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

If HWP has been already been enabled by BIOS, it may be
necessary to override some kernel command line parameters.
Once it has been enabled it requires a reset to be disabled.
Suggested-by: NRafael J. Wysocki <rafael@kernel.org>
Signed-off-by: NDoug Smythies <dsmythies@telus.net>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The current HWP calibration for hybrid processors in intel_pstate is
fragile, because it depends too much on the information provided by
the platform firmware via CPPC which may not be reliable enough.  It
also need not be so complicated.

In order to improve that mechanism and make it more resistant to
platform firmware issues, make it only use the CPPC nominal_perf
values to compute the HWP-to-frequency scaling factors for all
CPUs and possibly use the HWP_CAP highest_perf values to recompute
them if the ones derived from the CPPC nominal_perf values alone
appear to be too high.

Namely, fetch CPC.nominal_perf for all CPUs present in the system,
find the minimum one and use it as a reference for computing all of
the CPUs' scaling factors (using the observation that for the CPUs
having the minimum CPC.nominal_perf the HWP range of available
performance levels should be the same as the range of available
"legacy" P-states and so the HWP-to-frequency scaling factor for
them should be the same as the corresponding scaling factor used
for representing the P-state values in kHz).
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>
Tested-by: NZhang Rui <rui.zhang@intel.com>

Revert commit d0e936ad ("cpufreq: intel_pstate: Process HWP
Guaranteed change notification"), because it causes a NULL pointer
dereference to occur on Lenovo X1 gen9 laptops due to an HWP
guaranteed performance change interrupt arriving prematurely.

This feature will be revisited in the next cycle.
Reported-by: NJens Axboe <axboe@kernel.dk>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Introduce cpufreq HW driver which can support
CPU frequency adjust in MT6779 platform.
Signed-off-by: NHector.Yuan <hector.yuan@mediatek.com>
[ Viresh: Massaged the patch and cleaned some stuff. ]
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

This isn't used anymore, get rid of it.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The ->ready() callback is going away and since we don't do any important
stuff in sh_cpufreq_cpu_ready(), remove it.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The ready() callback was implemented earlier for acpi-cpufreq driver as
we wanted to use policy->cpuinfo.max_freq for which the policy was
required to be verified.

That is no longer the case and we can do the pr_warn() right from
->init() callback now. Remove acpi_cpufreq_cpu_ready().
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

As remote cpufreq updates are supported on QCOM platforms, set
dvfs_possible_from_any_cpu cpufreq driver flag.
Signed-off-by: NTaniya Das <tdas@codeaurora.org>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

The Qualcomm sa8155p, sm6350, sm8250 and sm8350 platforms also uses the
qcom-cpufreq-hw driver, so add them to the cpufreq-dt-platdev driver's
blocklist.
Signed-off-by: NBjorn Andersson <bjorn.andersson@linaro.org>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

Add interrupt support to notify the kernel of h/w initiated frequency
throttling by LMh. Convey this to scheduler via thermal presssure
interface.
Signed-off-by: NThara Gopinath <thara.gopinath@linaro.org>
[Viresh: Added changes for arch_topology.c to fix build errors ]
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

Set the newly added .register_em() callback to register with the EM
after the cpufreq policy is properly initialized.
Acked-by: NSudeep Holla <sudeep.holla@arm.com>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

Set the newly added .register_em() callback with
cpufreq_register_em_with_opp() to register with the EM core.
Acked-by: NSudeep Holla <sudeep.holla@arm.com>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

Set the newly added .register_em() callback with
cpufreq_register_em_with_opp() to register with the EM core.
Acked-by: NSudeep Holla <sudeep.holla@arm.com>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

It is possible that HWP guaranteed ratio is changed in response to
change in power and thermal limits. For example when Intel Speed Select
performance profile is changed or there is change in TDP, hardware can
send notifications. It is possible that the guaranteed ratio is
increased. This creates an issue when turbo is disabled, as the old
limits set in MSR_HWP_REQUEST are still lower and hardware will clip
to older limits.

This change enables HWP interrupt and process HWP interrupts. When
guaranteed is changed, calls cpufreq_update_policy() so that driver
callbacks are called to update to new HWP limits. This callback
is called from a delayed workqueue of 10ms to avoid frequent updates.
Signed-off-by: NSrinivas Pandruvada <srinivas.pandruvada@linux.intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

In the numa=off kernel command-line configuration init_chip_info() loops
around the number of chips and attempts to copy the cpumask of that node
which is NULL for all iterations after the first chip.

Hence, store the cpu mask for each chip instead of derving cpumask from
node while populating the "chips" struct array and copy that to the
chips[i].mask

Fixes: 053819e0 ("cpufreq: powernv: Handle throttling due to Pmax capping at chip level")
Cc: stable@vger.kernel.org # v4.3+
Reported-by: NShirisha Ganta <shirisha.ganta1@ibm.com>
Signed-off-by: NPratik R. Sampat <psampat@linux.ibm.com>
Reviewed-by: NGautham R. Shenoy <ego@linux.vnet.ibm.com>
[mpe: Rename goto label to out_free_chip_cpu_mask]
Signed-off-by: NMichael Ellerman <mpe@ellerman.id.au>
Link: https://lore.kernel.org/r/20210728120500.87549-2-psampat@linux.ibm.com

Set the newly added .register_em() callback with
cpufreq_register_em_with_opp() to register with the EM core.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

Set the newly added .register_em() callback with
cpufreq_register_em_with_opp() to register with the EM core.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

Set the newly added .register_em() callback with
cpufreq_register_em_with_opp() to register with the EM core.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

Set the newly added .register_em() callback with
cpufreq_register_em_with_opp() to register with the EM core.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

Set the newly added .register_em() callback with
cpufreq_register_em_with_opp() to register with the EM core.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

Many cpufreq drivers register with the energy model for each policy and
do exactly the same thing. Follow the footsteps of thermal-cooling, to
get it done from the cpufreq core itself.

Provide a new callback, which will be called, if present, by the cpufreq
core at the right moment (more on that in the code's comment). Also
provide a generic implementation that uses dev_pm_opp_of_register_em().

This also allows us to register with the EM at a later point of time,
compared to ->init(), from where the EM core can access cpufreq policy
directly using cpufreq_cpu_get() type of helpers and perform other work,
like marking few frequencies inefficient, this will be done separately.
Reviewed-by: NQuentin Perret <qperret@google.com>
Reviewed-by: NLukasz Luba <lukasz.luba@arm.com>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

Reuse the cpufreq core's registration of cooling device by setting the
CPUFREQ_IS_COOLING_DEV flag. Set this only if bL switcher isn't enabled.
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

The 1.2 GHz variant of the Armada 3720 SOC is unstable with DVFS: when
the SOC boots, the WTMI firmware sets clocks and AVS values that work
correctly with 1.2 GHz CPU frequency, but random crashes occur once
cpufreq driver starts scaling.

We do not know currently what is the reason:
- it may be that the voltage value for L0 for 1.2 GHz variant provided
  by the vendor in the OTP is simply incorrect when scaling is used,
- it may be that some delay is needed somewhere,
- it may be something else.

The most sane solution now seems to be to simply forbid the cpufreq
driver on 1.2 GHz variant.
Signed-off-by: NMarek Behún <kabel@kernel.org>
Fixes: 92ce45fb ("cpufreq: Add DVFS support for Armada 37xx")
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

The Qualcomm sm8150 platform uses the qcom-cpufreq-hw driver, so
add it to the cpufreq-dt-platdev driver's blocklist.
Signed-off-by: NThara Gopinath <thara.gopinath@linaro.org>
Reviewed-by: NBjorn Andersson <bjorn.andersson@linaro.org>
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>

The functions get_online_cpus() and put_online_cpus() have been
deprecated during the CPU hotplug rework. They map directly to
cpus_read_lock() and cpus_read_unlock().

Replace deprecated CPU-hotplug functions with the official version.
The behavior remains unchanged.
Signed-off-by: NSebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>