When the cpu enters idle, it stores the cpuidle state pointer in its
struct rq instance which in turn could be used to make a better decision
when balancing tasks.

As soon as the cpu exits its idle state, the struct rq reference is
cleared.

There are a couple of situations where the idle state pointer could be changed
while it is being consulted:

1. For x86/acpi with dynamic c-states, when a laptop switches from battery
   to AC that could result on removing the deeper idle state. The acpi driver
   triggers:
	'acpi_processor_cst_has_changed'
		'cpuidle_pause_and_lock'
			'cpuidle_uninstall_idle_handler'
				'kick_all_cpus_sync'.

All cpus will exit their idle state and the pointed object will be set to
NULL.

2. The cpuidle driver is unloaded. Logically that could happen but not
in practice because the drivers are always compiled in and 95% of them are
not coded to unregister themselves.  In any case, the unloading code must
call 'cpuidle_unregister_device', that calls 'cpuidle_pause_and_lock'
leading to 'kick_all_cpus_sync' as mentioned above.

A race can happen if we use the pointer and then one of these two scenarios
occurs at the same moment.

In order to be safe, the idle state pointer stored in the rq must be
used inside a rcu_read_lock section where we are protected with the
'rcu_barrier' in the 'cpuidle_uninstall_idle_handler' function. The
idle_get_state() and idle_put_state() accessors should be used to that
effect.
Signed-off-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
Signed-off-by: NNicolas Pitre <nico@linaro.org>
Signed-off-by: NPeter Zijlstra (Intel) <peterz@infradead.org>
Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net>
Cc: linux-pm@vger.kernel.org
Cc: linaro-kernel@lists.linaro.org
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Link: http://lkml.kernel.org/n/tip-@git.kernel.orgSigned-off-by: NIngo Molnar <mingo@kernel.org>

Currently kick_all_cpus_sync() or smp_call_function() can not
break the polling idle cpu immediately.

Instead using wake_up_all_idle_cpus() which can wake up the polling idle
cpu quickly is much more helpful for power.
Signed-off-by: NChuansheng Liu <chuansheng.liu@intel.com>
Signed-off-by: NPeter Zijlstra (Intel) <peterz@infradead.org>
Cc: linux-pm@vger.kernel.org
Cc: changcheng.liu@intel.com
Cc: xiaoming.wang@intel.com
Cc: souvik.k.chakravarty@intel.com
Cc: luto@amacapital.net
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: linux-pm@vger.kernel.org
Link: http://lkml.kernel.org/r/1409815075-4180-3-git-send-email-chuansheng.liu@intel.comSigned-off-by: NIngo Molnar <mingo@kernel.org>

Commit af040ffc ("ARM: make it easier to check the CPU part number
correctly") changed ARM_CPU_PART_X masks, and the way they are returned and
checked against. Usage of read_cpuid_part_number() is now deprecated, and
calling places updated accordingly. This actually broke cpuidle-big_little
initialization, as bl_idle_driver_init() performs a check using an hardcoded
mask on cpu_id.

Create an interface to perform the check (that is now even easier to read).
Define also a proper mask (ARM_CPU_PART_MASK) that makes this kind of checks
cleaner and helps preventing bugs in the future. Update usage accordingly.
Signed-off-by: NJuri Lelli <juri.lelli@arm.com>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Due to recent consolidation of Exynos suspend and cpuidle code, some
parts of suspend and resume sequences are executed two times, once from
exynos_pm_syscore_ops and then from exynos_cpu_pm_notifier() and thus it
breaks suspend, at least on Exynos4-based boards. In addition, simple
core power down from a cpuidle driver could, in case of CPU 0 could
result in calling functions that are specific to suspend and deeper idle
states.

This patch fixes the issue by moving those operations outside the CPU PM
notifier into suspend and AFTR code paths. This leads to a bit of code
duplication, but allows additional code simplification, so in the end
more code is removed than added.

Fixes: 85f9f908 ("ARM: EXYNOS: Use the cpu_pm notifier for pm")
Cc: Kukjin Kim <kgene.kim@samsung.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Olof Johansson <olof@lixom.net>
Cc: arm@kernel.org
Signed-off-by: NTomasz Figa <t.figa@samsung.com>
[b.zolnierkie: ported patch over current changes]
[b.zolnierkie: fixed exynos_aftr_finisher() return value]
Signed-off-by: NBartlomiej Zolnierkiewicz <b.zolnierkie@samsung.com>
Signed-off-by: NOlof Johansson <olof@lixom.net>

The menu governer makes separate lookups of the CPU runqueue to get
load and number of IO waiters but it can be done with a single lookup.
Signed-off-by: NMel Gorman <mgorman@suse.de>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

menu_select() via inline functions calls nr_iowait_cpu() twice as much
as necessary.
Signed-off-by: NMel Gorman <mgorman@suse.de>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The ktime_to_us implementation is slightly better than the one implemented
in menu.c. Use it
Signed-off-by: NMel Gorman <mgorman@suse.de>
Acked-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

We use do_div even though the divisor will usually be a power-of-two
unless there are unusual outliers. Use shifts where possible
Signed-off-by: NMel Gorman <mgorman@suse.de>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Flags from device-tree need to be parsed with accessors for
interpreting correct value in little-endian.
Signed-off-by: NVaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
Reviewed-by: NPreeti U. Murthy <preeti@linux.vnet.ibm.com>
CC: <stable@vger.kernel.org>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

The time measurement is already done in the cpuidle framework in the
'cpuidle_enter_state' function.
Signed-off-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Like the coupled idle state, it is not up to the user to set this option
but the driver to select it.

Remove the interactive selection of this option.
Signed-off-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

use CPUIDLE_DRIVER_STATE_START, instead of hardcoded value 0. As,
CPUIDLE_DRIVER_STATE_START can be 1/0 based on
CONFIG_ARCH_HAS_CPU_RELAX is defined or not.
Signed-off-by: NMohammad Merajul Islam Molla <meraj.enigma@gmail.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

big_little CPU idle driver references functions defined in MCPM driver.
Thus make it depend on MCPM to avoid the following errors:

drivers/built-in.o: In function `bl_enter_powerdown':
drivers/cpuidle/cpuidle-big_little.c:134: undefined reference to `mcpm_cpu_powered_up'
drivers/built-in.o: In function `bl_powerdown_finisher':
drivers/cpuidle/cpuidle-big_little.c:104: undefined reference to `mcpm_set_entry_vector'
drivers/cpuidle/cpuidle-big_little.c:111: undefined reference to `mcpm_cpu_suspend'
make: *** [vmlinux] Error 1
Reported-by: NAndreas Färber <afaerber@suse.de>
Signed-off-by: NSachin Kamat <sachin.kamat@samsung.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

STDDEV_THRESH was once defined and used in menu governor. But now its no longer
used anywhere. So removing the define.
Signed-off-by: NMohammad Merajul Islam Molla <meraj.enigma@gmail.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

This commit adds the list of cpuidle states supported by the Armada
38x SoC in the cpuidle-mvebu-v7 driver, as well as the necessary logic
around it to support this SoC.
Signed-off-by: NThomas Petazzoni <thomas.petazzoni@free-electrons.com>
Acked-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
Link: https://lkml.kernel.org/r/1406120453-29291-14-git-send-email-thomas.petazzoni@free-electrons.comSigned-off-by: NJason Cooper <jason@lakedaemon.net>

This commit adds the list of cpuidle states supported by the Armada
370 SoC in the cpuidle-mvebu-v7 driver, as well as the necessary logic
around it to support this SoC.
Signed-off-by: NThomas Petazzoni <thomas.petazzoni@free-electrons.com>
Acked-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
Link: https://lkml.kernel.org/r/1406120453-29291-13-git-send-email-thomas.petazzoni@free-electrons.comSigned-off-by: NJason Cooper <jason@lakedaemon.net>

This driver will be able to manage the cpuidle for more SoCs than just
Armada 370 and XP. It will also support Armada 38x and potentially
other SoC of the Marvell Armada EBU family. To take this into account,
this patch renames the driver and its symbols.

It also changes the driver name from cpuidle-armada-370-xp to
cpuidle-armada-xp, because separate platform drivers will be
registered for the other SoC types. This change must be done
simultaneously in the cpuidle driver and in the PMSU code in order to
remain bisectable.
Signed-off-by: NGregory CLEMENT <gregory.clement@free-electrons.com>
Signed-off-by: NThomas Petazzoni <thomas.petazzoni@free-electrons.com>
Acked-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
Link: https://lkml.kernel.org/r/1406120453-29291-12-git-send-email-thomas.petazzoni@free-electrons.comSigned-off-by: NJason Cooper <jason@lakedaemon.net>

cpuidle driver name sysfs node is read-only, so permissions should be 0444.
Signed-off-by: NMohammad Merajul Islam Molla <meraj.enigma@gmail.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Add "samsung,exynos5420" compatible string to initialize generic
big-little cpuidle driver for Exynos5420.
Signed-off-by: NChander Kashyap <chander.kashyap@linaro.org>
Reviewed-by: NTomasz Figa <t.figa@samsung.com>
Acked-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
Signed-off-by: NKukjin Kim <kgene.kim@samsung.com>

Add support to select generic big-little cpuidle driver for Samsung Exynos
series SoC's. This is required for Exynos big-llittle SoC's eg, Exynos5420.
Signed-off-by: NChander Kashyap <chander.kashyap@linaro.org>
Reviewed-by: NTomasz Figa <t.figa@samsung.com>
Acked-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
Signed-off-by: NKukjin Kim <kgene.kim@samsung.com>

This driver will be used by many big.Little Soc's. As of now it does
string matching of hardcoded compatible string to init the driver. This
comparison list will keep on growing with addition of new SoC's.
Hence add of_device_id structure to collect the compatible strings of
SoC's using this driver.
Signed-off-by: NChander Kashyap <chander.kashyap@linaro.org>
Reviewed-by: NTomasz Figa <t.figa@samsung.com>
Acked-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
Signed-off-by: NKukjin Kim <kgene.kim@samsung.com>

idle_exit event is the first event after a core exits
idle state. So this should be traced before local irq
is ebabled. Likewise idle_entry is the last event before
a core enters idle state. This will ease visualising the
cpu idle state from kernel traces.
Signed-off-by: NSandeep Tripathy <sandeep.tripathy@linaro.org>
Acked-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
[rjw: Subject, rebase]
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Signed-off-by: NAntonio Ospite <ao2@ao2.it>
Cc: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
Cc: "tuukka.tikkanen@linaro.org" <tuukka.tikkanen@linaro.org>
Signed-off-by: NJiri Kosina <jkosina@suse.cz>

The length name of the states 2 was too long to fit in the allocated
string (limited to 16 bytes). This lead to improper string displayed
through sysfs.

This patch shorten the name by removing the reference to Marvell and
to the CPU as both are implicit. For coherency the same change have
been done for the states 1.
Signed-off-by: NGregory CLEMENT <gregory.clement@free-electrons.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Currently when entering fastsleep we clear all LPCR PECE bits.

This patch changes it to only clear the decrementer bit (ie. PECE1), which is
the only bit we really need to clear here.  This is needed if we want to set
other wakeup causes like the PECEDH bit so we can use hypervisor doorbells on
powernv.  Also we no longer clear the MER bit as it should never be set in the
host anyway.
Signed-off-by: NMichael Neuling <mikey@neuling.org>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

poll_idle is the archetypal polling idle loop; tell the core idle
code about it.

This avoids pointless IPIs when all of the other cpuidle states are
disabled.
Signed-off-by: NAndy Lutomirski <luto@amacapital.net>
Signed-off-by: NPeter Zijlstra <peterz@infradead.org>
Cc: nicolas.pitre@linaro.org
Cc: umgwanakikbuti@gmail.com
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: linux-kernel@vger.kernel.org
Cc: linux-pm@vger.kernel.org
Link: http://lkml.kernel.org/r/c65ce49615d338bae8fb79df5daffab19353c900.1401902905.git.luto@amacapital.netSigned-off-by: NIngo Molnar <mingo@kernel.org>

This patch adds a cpuidle driver for systems based around the MIPS
Coherent Processing System (CPS) architecture. It supports four idle
states:

  - The standard MIPS wait instruction.

  - The non-coherent wait, clock gated & power gated states exposed by
    the recently added pm-cps layer.

The pm-cps layer is used to enter all the deep idle states. Since cores
in the clock or power gated states cannot service interrupts, the
gic_send_ipi_single function is modified to send a power up command for
the appropriate core to the CPC in cases where the target CPU has marked
itself potentially incoherent.
Signed-off-by: NPaul Burton <paul.burton@imgtec.com>

Signed-off-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
Reviewed-by: NViresh Kumar <viresh.kumar@linaro.org>
Reviewed-by: NBartlomiej Zolnierkiewicz <b.zolnierkie@samsung.com>
Reviewed-by: NTomasz Figa <t.figa@samsung.com>
Signed-off-by: NKukjin Kim <kgene.kim@samsung.com>

Add the wfi, cpu idle and cpu deep idle power states support for the
Armada XP SoCs.

All the latencies and the power consumption values used at the
"armada_370_xp_idle_driver" structure are preliminary and will be
modified in the future after running some measurements and analysis.

Based on the work of Nadav Haklai.
Signed-off-by: NNadav Haklai <nadavh@marvell.com>
Link: https://lkml.kernel.org/r/1397488214-20685-11-git-send-email-gregory.clement@free-electrons.comSigned-off-by: NGregory CLEMENT <gregory.clement@free-electrons.com>
Link: https://lkml.kernel.org/r/1397488214-20685-11-git-send-email-gregory.clement@free-electrons.comAcked-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
Signed-off-by: NJason Cooper <jason@lakedaemon.net>

If freeze_enter() is called, we want to bypass the current cpuidle
governor and always use the deepest available (that is, not disabled)
C-state, because we want to save as much energy as reasonably possible
then and runtime latency constraints don't matter at that point, since
the system is in a sleep state anyway.
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>
Tested-by: NAubrey Li <aubrey.li@linux.intel.com>

In menu_select function we check for correction factor every time.
If it is zero we are initializing to unity. Hence move it to init function
and initialise by unity, hence avoid repeated comparisons.
Signed-off-by: NChander Kashyap <chander.kashyap@linaro.org>
Reviewed-by: NTuukka Tikkanen <tuukka.tikkanen@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

If there is a PM QoS latency limit and all of the sufficiently shallow
C-states are disabled, the cpuidle menu governor returns 0 which on
some systems is CPUIDLE_DRIVER_STATE_START and shouldn't be returned
if that C-state has been disabled.

Fix the issue by modifying the menu governor to return (-1) in such
situations.
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Since both cpuidle_enabled() and cpuidle_select() are only called by
cpuidle_idle_call(), it is not really useful to keep them separate
and combining them will help to avoid complicating cpuidle_idle_call()
even further if governors are changed to return error codes sometimes.

This code modification shouldn't lead to any functional changes.
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Add cpuidle support for ARM Cirrus Logic CLPS711X CPUs.
This CPU has an unique internal register and write to this location
will put the system into the Idle State by halting the clock to the
processor until an interrupt is generated.
Signed-off-by: NAlexander Shiyan <shc_work@mail.ru>
Signed-off-by: NDaniel Lezcano <daniel.lezcano@linaro.org>

Mostly scripted conversion of the smp_mb__* barriers.
Signed-off-by: NPeter Zijlstra <peterz@infradead.org>
Acked-by: NPaul E. McKenney <paulmck@linux.vnet.ibm.com>
Link: http://lkml.kernel.org/n/tip-55dhyhocezdw1dg7u19hmh1u@git.kernel.org
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: linux-arch@vger.kernel.org
Signed-off-by: NIngo Molnar <mingo@kernel.org>

From user space, there is no way to know the target residency for each idle
state. If we want to write tools to measure the accuracy of the idle state
selection from the governor, we need this info.

As the exit latency is exported through sysfs, exporting the target residency
in the same place makes sense.
Signed-off-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

As described by a comment at the end of cpuidle_enter_state_coupled it
can be inefficient for coupled idle states to return with IRQs enabled
since they may proceed to service an interrupt instead of clearing the
coupled idle state. Until they have finished & cleared the idle state
all CPUs coupled with them will spin rather than being able to enter a
safe idle state.

Commits e1689795 "cpuidle: Add common time keeping and irq
enabling" and 554c06ba "cpuidle: remove en_core_tk_irqen flag" led
to the cpuidle_enter_state enabling interrupts for all idle states,
including coupled ones, making this inefficiency unavoidable by drivers
& the local_irq_enable near the end of cpuidle_enter_state_coupled
redundant. This patch avoids enabling interrupts in cpuidle_enter_state
after a coupled state has been entered, allowing them to remain disabled
until all coupled CPUs have exited the idle state and
cpuidle_enter_state_coupled re-enables them.

Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Signed-off-by: NPaul Burton <paul.burton@imgtec.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The cpuidle_idle_call does nothing more than calling the three individuals
function and is no longer used by any arch specific code but only in the
cpuidle framework code.

We can move this function into the idle task code to ensure better
proximity to the scheduler code.
Signed-off-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
Acked-by: NNicolas Pitre <nicolas.pitre@linaro.org>
Signed-off-by: NPeter Zijlstra <peterz@infradead.org>
Cc: rjw@rjwysocki.net
Cc: preeti@linux.vnet.ibm.com
Link: http://lkml.kernel.org/r/1393832934-11625-2-git-send-email-daniel.lezcano@linaro.orgSigned-off-by: NIngo Molnar <mingo@kernel.org>

In order to allow better integration between the cpuidle framework and the
scheduler, reducing the distance between these two sub-components will
facilitate this integration by moving part of the cpuidle code in the idle
task file and, because idle.c is in the sched directory, we have access to
the scheduler's private structures.

This patch splits the cpuidle_idle_call main entry function into 3 calls
to a newly added API:

 1. select the idle state
 2. enter the idle state
 3. reflect the idle state

The cpuidle_idle_call calls these three functions to implement the main
idle entry function.
Signed-off-by: NDaniel Lezcano <daniel.lezcano@linaro.org>
Acked-by: NNicolas Pitre <nico@linaro.org>
Signed-off-by: NPeter Zijlstra <peterz@infradead.org>
Cc: rjw@rjwysocki.net
Cc: preeti@linux.vnet.ibm.com
Link: http://lkml.kernel.org/r/1393832934-11625-1-git-send-email-daniel.lezcano@linaro.orgSigned-off-by: NIngo Molnar <mingo@kernel.org>

For some platforms, a poll state is inserted in the cpuidle driver states.
The flags for the state do not indicate that timekeeping is not affected.
As the state does not do anything apart from calling cpu_relax(), the
times returned by ktime_get should remain valid. Add the missing flag.
Signed-off-by: NTuukka Tikkanen <tuukka.tikkanen@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>