Denverton is an Intel Atom based micro server which shares the same
Goldmont architecture as Broxton. The available C-states on
Denverton is a subset of Broxton with only C1, C1e, and C6.
Signed-off-by: NJacob Pan <jacob.jun.pan@linux.intel.com>
Signed-off-by: NLen Brown <len.brown@intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The Kconfig for this driver is currently declared with:

config INTEL_IDLE
        bool "Cpuidle Driver for Intel Processors"

...meaning that it currently is not being built as a module by anyone.

This was done in commit 6ce9cd86
("intel_idle: disable module support") since "...the module capability
is cauing more trouble than it is worth."

This was done over 5y ago, and Daniel adds that:

    ...the modular support has been removed from almost all the cpuidle
    drivers and the cpuidle framework is no longer assuming driver could
    be unloaded.

    Removing the modular dead code in the driver makes sense as this
    what have been done in the others drivers.

So lets remove the modular code that is essentially orphaned, so that
when reading the driver there is no doubt it is builtin-only.

Since module_init translates to device_initcall in the non-modular
case, the init ordering remains unchanged with this commit.  At a
later date we might want to consider whether subsys_init or another
init category seems more appropriate than device_init.

We replace module.h with moduleparam.h since the file does declare
some module parameters, and leaving them as such is currently the
easiest way to remain compatible with existing boot arg use cases.

Note that MODULE_DEVICE_TABLE is a no-op for non-modular code.

Also note that we can't remove intel_idle_cpuidle_devices_uninit() as
that is still used for unwind purposes if the init fails.

We also delete the MODULE_LICENSE tag etc. since all that information
is already contained at the top of the file in the comments.
Signed-off-by: NPaul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: NLen Brown <len.brown@intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Use the new INTEL_FAM6_* macros for intel_idle.c.  Also fix up
some of the macros to be consistent with how some of the
intel_idle code refers to the model.

There's on oddity here: model 0x1F is uniquely referred to here
and nowhere else that I could find.  0x1E/0x1F are just spelled
out as "Intel Core i7 and i5 Processors" in the SDM or as "Intel
processors based on the Nehalem, Westmere microarchitectures" in
the RDPMC section.  Comments between tables 19-19 and 19-20 in
the SDM seem to point to 0x1F being some kind of Westmere, so
let's call it "WESTMERE2".
Signed-off-by: NDave Hansen <dave.hansen@linux.intel.com>
Acked-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Dave Hansen <dave@sr71.net>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Len Brown <lenb@kernel.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: jacob.jun.pan@intel.com
Cc: linux-pm@vger.kernel.org
Link: http://lkml.kernel.org/r/20160603001932.EE978EB9@viggo.jf.intel.comSigned-off-by: NIngo Molnar <mingo@kernel.org>

Broxton has all the HSW C-states, except C3.
BXT C-state timing is slightly different.

Here we trust the IRTL MSRs as authority
on maximum C-state latency, and override the driver's tables
with the values found in the associated IRTL MSRs.
Further we set the target_residency to 1x maximum latency,
trusting the hardware demotion logic.
Signed-off-by: NLen Brown <len.brown@intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

KBL is similar to SKL
Signed-off-by: NLen Brown <len.brown@intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

SKX is similar to BDX
Signed-off-by: NLen Brown <len.brown@intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

This driver registers cpuidle devices when a CPU comes online, but it
leaves the registrations in place when a CPU goes offline.  The module
exit code only unregisters the currently online CPUs, leaving the
devices for offline CPUs dangling.

This patch changes the driver to clean up all registrations on exit,
even those from CPUs that are offline.
Signed-off-by: NRichard Cochran <rcochran@linutronix.de>
Signed-off-by: NLen Brown <len.brown@intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

If a cpuidle registration error occurs during the hot plug notifier
callback, we should really inform the hot plug machinery instead of
just ignoring the error.  This patch changes the callback to properly
return on error.
Signed-off-by: NRichard Cochran <rcochran@linutronix.de>
Signed-off-by: NLen Brown <len.brown@intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The helper function, intel_idle_cpu_init, registers one new device
with the cpuidle layer.  If the registration should fail, that
function immediately calls intel_idle_cpuidle_devices_uninit() to
unregister every last CPU's device.  However, it makes no sense to do
so, when called from the hot plug notifier callback.

This patch moves the call to intel_idle_cpuidle_devices_uninit()
outside of the helper function to the one call site that actually
needs to perform the de-registrations.
Signed-off-by: NRichard Cochran <rcochran@linutronix.de>
Signed-off-by: NLen Brown <len.brown@intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

This driver sets the broadcast tick quite early on during probe and does
not clean up again in cast of failure.  This patch moves the setup call
after the registration, placing the on_each_cpu() calls within the global
CPU lock region.
Signed-off-by: NRichard Cochran <rcochran@linutronix.de>
Signed-off-by: NLen Brown <len.brown@intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The helper function, intel_idle_cpuidle_devices_uninit, frees the
globally allocated per-CPU data.  However, this function is invoked
from the hot plug notifier callback at a time when freeing that data
is not safe.

If the call to cpuidle_register_driver() should fail (say, due to lack
of memory), then the driver will free its per-CPU region.  On the
*next* CPU_ONLINE event, the driver will happily use the region again
and even free it again if the failure repeats.

This patch fixes the issue by moving the call to free_percpu() outside
of the helper function at the two call sites that actually need to
free the per-CPU data.
Signed-off-by: NRichard Cochran <rcochran@linutronix.de>
Signed-off-by: NLen Brown <len.brown@intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

In the module_init() method, if the per-CPU allocation fails, then the
active cpuidle registration is not cleaned up.  This patch fixes the
issue by attempting the allocation before registration, and then
cleaning it up again on registration failure.
Signed-off-by: NRichard Cochran <rcochran@linutronix.de>
Signed-off-by: NLen Brown <len.brown@intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

In the module_exit() method, this driver first frees its per-CPU
pointer, then unregisters a callback making use of the pointer.
Furthermore, the function, intel_idle_cpuidle_devices_uninit, is racy
against CPU hot plugging as it calls for_each_online_cpu().

This patch corrects the issues by unregistering first on the exit path
while holding the hot plug lock.
Signed-off-by: NRichard Cochran <rcochran@linutronix.de>
Signed-off-by: NLen Brown <len.brown@intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The function, intel_idle_cpuidle_driver_init, makes calls on each CPU
to auto_demotion_disable() and c1e_promotion_disable().  These calls
are redundant, as intel_idle_cpu_init() does the same calls just a bit
later on.  They are also premature, as the driver registration may yet
fail.

This patch removes the redundant code.
Signed-off-by: NRichard Cochran <rcochran@linutronix.de>
Signed-off-by: NLen Brown <len.brown@intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The function, intel_idle_cpuidle_driver_init, delivers no error codes
at all.  This patch changes the function to return 'void' instead of
returning zero.
Signed-off-by: NRichard Cochran <rcochran@linutronix.de>
Signed-off-by: NLen Brown <len.brown@intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Signed-off-by: NRichard Cochran <rcochran@linutronix.de>
Signed-off-by: NLen Brown <len.brown@intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Enables "Intel(R) Xeon Phi(TM) Processor x200 Product Family" support,
formerly code-named KNL. It is based on modified Intel Atom Silvermont
microarchitecture.
Signed-off-by: NDasaratharaman Chandramouli <dasaratharaman.chandramouli@intel.com>
[micah.barany@intel.com: adjusted values of residency and latency]
Signed-off-by: NMicah Barany <micah.barany@intel.com>
[hubert.chrzaniuk@intel.com: removed deprecated CPUIDLE_FLAG_TIME_VALID flag]
Signed-off-by: NHubert Chrzaniuk <hubert.chrzaniuk@intel.com>
Signed-off-by: NPawel Karczewski <pawel.karczewski@intel.com>
Signed-off-by: NLen Brown <len.brown@intel.com>

Some SKL-H configurations require "intel_idle.max_cstate=7" to boot.
While that is an effective workaround, it disables C10.

This patch detects the problematic configuration,
and disables C8 and C9, keeping C10 enabled.

Note that enabling SGX in BIOS SETUP can also prevent this issue,
if the system BIOS provides that option.

https://bugzilla.kernel.org/show_bug.cgi?id=109081
"Freezes with Intel i7 6700HQ (Skylake), unless intel_idle.max_cstate=7"
Signed-off-by: NLen Brown <len.brown@intel.com>
Cc: stable@vger.kernel.org

Addition of PC9 state, and minor tweaks to existing PC6 and PC8 states.
Signed-off-by: NLen Brown <len.brown@intel.com>

Skylake Client CPU idle Power states (C-states)
are similar to the previous generation, Broadwell.
However, Skylake does get its own table with updated
worst-case latency and average energy-break-even residency values.
Signed-off-by: NLen Brown <len.brown@intel.com>

intel_idle uses a NULL "enter" field in a cpuidle state
to recognize the invalid entry terminating a variable-length array.

Linux-4.0 added support for the system-wide "freeze" state
in cpuidle drivers via the new "enter_freeze" field.

The natural way to expose a deep idle state for freeze,
but not for run-time idle is to supply "enter_freeze" without "enter";
so we update the driver to accept such states.
Signed-off-by: NLen Brown <len.brown@intel.com>

The CPU ids are only tested in intel_idle_probe() which is itself an
__init function. For the MODULE_DEVICE_TABLE() file2alias doesn't care
about the section, just about the symbol name. So it's safe to mark
the cpu id array as  __initconst so its memory can be released after
initialization is done.
Signed-off-by: NMathias Krause <minipli@googlemail.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Replace the clockevents_notify() call with an explicit function call.
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>
Cc: Len Brown <lenb@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20714596.QMfNNPbuyU@vostro.rjw.lanSigned-off-by: NIngo Molnar <mingo@kernel.org>

Replace the clockevents_notify() call with an explicit function call.
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>
Cc: Len Brown <lenb@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/3878165.rXNXrtVNuy@vostro.rjw.lanSigned-off-by: NIngo Molnar <mingo@kernel.org>

Support C-states for the Airmont core in the Cherrytrail and Braswell SOCs.
The states are similar to those of Silvermont in Baytrail,
except both flavors of C6 states are faster.
Signed-off-by: NLen Brown <len.brown@intel.com>
Cc: Kumar P Mahesh <mahesh.kumar.p@intel.com>
Cc: Alan Cox <alan@linux.intel.com>
Cc: Mika Westerberg <mika.westerberg@linux.intel.com>

On some Silvermont-Core/Baytrail-SOC systems,
C1E latency is higher than original specifications.
Although C1E is still enumerated in CPUID.MWAIT.EDX,
we delete the state from intel_idle to avoid latency impact.

Under some conditions, the latency of the C6N-BYT and C6S-BYT states
may exceed the specified values of 40 and 140 usec, respectively.
Increase those values to 300 and 500 usec; to assure
that the hardware does not violate constraints that may be set
by the Linux PM_QOS sub-system.

Also increase the C7-BYT target residency to 4.0 ms from 1.5 ms.
Signed-off-by: NLen Brown <len.brown@intel.com>
Cc: Kumar P Mahesh <mahesh.kumar.p@intel.com>
Cc: Alan Cox <alan@linux.intel.com>
Cc: Mika Westerberg <mika.westerberg@linux.intel.com>
Cc: <stable@vger.kernel.org>

Add an ->enter_freeze callback routine, intel_idle_freeze(), to
the intel_idle driver and point the ->enter_freeze callback
pointers of all of the driver's state objects to it.
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: NPeter Zijlstra (Intel) <peterz@infradead.org>

Signed-off-by: NLen Brown <len.brown@intel.com>

The only place where the time is invalid is when the ACPI_CSTATE_FFH entry
method is not set. Otherwise for all the drivers, the time can be correctly
measured.

Instead of duplicating the CPUIDLE_FLAG_TIME_VALID flag in all the drivers
for all the states, just invert the logic by replacing it by the flag
CPUIDLE_FLAG_TIME_INVALID, hence we can set this flag only for the acpi idle
driver, remove the former flag from all the drivers and invert the logic with
this flag in the different governor.
Signed-off-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Broadwell (BDW) is similar to Haswell (HSW), the preceding processor generation.

Currently, the only difference in their C-state tables is that PC3 max exit latency
is 33usec on HSW and 40usec on BDW.
Signed-off-by: NLen Brown <len.brown@intel.com>

Power efficiency improves on Baytrail (Intel Atom Processor E3000)
when Linux disables C6 auto-demotion.

Based on work by Srinidhi Kasagar <srinidhi.kasagar@intel.com>.
Signed-off-by: NLen Brown <len.brown@intel.com>
Cc: x86@kernel.org

Ivy Town idle state table will not be set as intended. Fix it.

Picked up by Coverity - CID 1201420/1201421.

Fixes: 0138d8f0 ("intel_idle: fine-tune IVT residency targets")
Signed-off-by: NChristoph Jaeger <christophjaeger@linux.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Ivy Town processors have slightly different properties
than Ivy Bridge processors, particuarly as socket count grows.
Here we add dedicated tables covering 1-2 socket,
3-4 socket, and > 4 socket IVT configurations.

This reduces the frequency of deep transitions on those systems,
which can impact throughput.
Signed-off-by: NLen Brown <len.brown@intel.com>

Subsystems that want to register CPU hotplug callbacks, as well as perform
initialization for the CPUs that are already online, often do it as shown
below:

	get_online_cpus();

	for_each_online_cpu(cpu)
		init_cpu(cpu);

	register_cpu_notifier(&foobar_cpu_notifier);

	put_online_cpus();

This is wrong, since it is prone to ABBA deadlocks involving the
cpu_add_remove_lock and the cpu_hotplug.lock (when running concurrently
with CPU hotplug operations).

Instead, the correct and race-free way of performing the callback
registration is:

	cpu_notifier_register_begin();

	for_each_online_cpu(cpu)
		init_cpu(cpu);

	/* Note the use of the double underscored version of the API */
	__register_cpu_notifier(&foobar_cpu_notifier);

	cpu_notifier_register_done();

Fix the intel-idle code by using this latter form of callback registration.

Cc: Len Brown <lenb@kernel.org>
Cc: Ingo Molnar <mingo@kernel.org>
Signed-off-by: NSrivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Add CPU ID for Atom N2600/N2800 processors. Datasheets indicate support
for this, detailed information about potential quirks or limitations are
missing, though. So we just reuse the definition for the previous ATOM
series. Tests on N2800 systems showed that this addition is fine an can
reduce power consumption by about 0.25 W (personally confirmed on Intel
DN2800MT).
Signed-off-by: NJan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: NLen Brown <len.brown@intel.com>

Bay Trail (BYT) is a family of Silvermont-core Atom Processor SOCs,
including the Intel Atom Processor Z36xxx and Z37xxx Series.

Although it shares the Silvermont core with Avoton,
BYT is optimized for mobile, and thus it supports
different power saving CPU idle states.

Note that not all versions of Bay Trail HW support all
of the states listed in the driver.
Signed-off-by: NLen Brown <len.brown@intel.com>
Tested-by: NAubrey Li <aubrey.li@linux.intel.com>

Like acpi_idle, intel_idle compared sub-state numbers
to the number of supported sub-states -- discarding
sub-states numbers that were numbered >= the number of states.

But some Bay Trail SOCs use sparse sub-state numbers,
so we can't make such a comparison if we are going
to access those states.

So now we simply check that _some_ sub-states are
supported for the given state, and assume that the
sub-state number in our driver is valid.

In practice, the driver is correct, and even if it were not,
the hardware clips invalid sub-state requests to valid ones.

No entries in the driver require this change,
but Bay Trail will need it.
Signed-off-by: NLen Brown <len.brown@intel.com>

intel_idle driver sets dev->state_count to drv->state_count so
the default dev->state_count initialization in cpuidle_enable_device()
(called from cpuidle_register_device()) can be used instead.
Signed-off-by: NBartlomiej Zolnierkiewicz <b.zolnierkie@samsung.com>
Signed-off-by: NKyungmin Park <kyungmin.park@samsung.com>
Cc: Len Brown <lenb@kernel.org>
Reviewed-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

If the system is booted with some CPUs offline C1E promotion disable quirk
won't be applied because on_each_cpu() in intel_idle_cpuidle_driver_init()
operates only on online CPUs. Fix it by adding the C1E promotion disable
handling to intel_idle_cpu_init() (which is also called during CPU_ONLINE
operation).
Signed-off-by: NBartlomiej Zolnierkiewicz <b.zolnierkie@samsung.com>
Signed-off-by: NKyungmin Park <kyungmin.park@samsung.com>
Cc: Len Brown <lenb@kernel.org>
Reviewed-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Close avn_cstates array with correct marker to avoid overflow
in function intel_idle_cpu_init().

[rjw: The problem was introduced when commit 22e580d0 was merged
 on top of eba682a5 (intel_idle: shrink states tables).]

Fixes: 22e580d0 (intel_idle: Fixed C6 state on Avoton/Rangeley processors)
Signed-off-by: NJiang Liu <jiang.liu@linux.intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>