Add a sched_clock() implementation to Versatile Express using the new
sched_clock() infrastructure for extending 32bit counters to full
64-bit nanoseconds.
Tested-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Convert versatile platforms to use the new sched_clock() infrastructure
for extending 32bit counters to full 64-bit nanoseconds.
Tested-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Convert orion platforms to use the new sched_clock() infrastructure for
extending 32bit counters to full 64-bit nanoseconds.
Acked-by: NNicolas Pitre <nico@fluxnic.net>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Convert omap to use the new sched_clock() infrastructure for extending
32bit counters to full 64-bit nanoseconds.
Tested-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Convert nomadik platforms to use the new sched_clock() infrastructure
for extending 32bit counters to full 64-bit nanoseconds.
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Convert iop platforms to use the new sched_clock() infrastructure for
extending 32bit counters to full 64-bit nanoseconds.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Convert u300 to use the new sched_clock() infrastructure for extending
32bit counters to full 64-bit nanoseconds.
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Convert tegra to use the new sched_clock() infrastructure for extending
32bit counters to full 64-bit nanoseconds.
Tested-by: NOlof Johansson <olof@lixom.net>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Convert sa1100 to use the new sched_clock() infrastructure for extending
32bit counters to full 64-bit nanoseconds.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Convert pxa to use the new sched_clock() infrastructure for extending
32bit counters to full 64-bit nanoseconds.
Tested-by: NEric Miao <eric.y.miao@gmail.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Convert mmp to use the new sched_clock() infrastructure for extending
32bit counters to full 64-bit nanoseconds.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Convert ixp4xx to use the new sched_clock() infrastructure for
extending 32bit counters to full 64-bit nanoseconds.
Tested-by: NMikael Pettersson <mikpe@it.uu.se>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Provide common sched_clock() infrastructure for platforms to use to
create a 64-bit ns based sched_clock() implementation from a counter
running at a non-variable clock rate.

This implementation is based upon maintaining an epoch for the counter
and an epoch for the nanosecond time.  When we desire a sched_clock()
time, we calculate the number of counter ticks since the last epoch
update, convert this to nanoseconds and add to the epoch nanoseconds.

We regularly refresh these epochs within the counter wrap interval.
We perform a similar calculation as above, and store the new epochs.

We read and write the epochs in such a way that sched_clock() can easily
(and locklessly) detect when an update is in progress, and repeat the
loading of these constants when they're known not to be stable.  The
one caveat is that sched_clock() is not called in the middle of an
update.  We achieve that by disabling IRQs.

Finally, if the clock rate is known at compile time, the counter to ns
conversion factors can be specified, allowing sched_clock() to be tightly
optimized.  We ensure that these factors are correct by providing an
initialization function which performs a run-time check.
Acked-by: NPeter Zijlstra <peterz@infradead.org>
Tested-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Tested-by: NWill Deacon <will.deacon@arm.com>
Tested-by: NMikael Pettersson <mikpe@it.uu.se>
Tested-by: NEric Miao <eric.y.miao@gmail.com>
Tested-by: NOlof Johansson <olof@lixom.net>
Tested-by: NJamie Iles <jamie@jamieiles.com>
Reviewed-by: NNicolas Pitre <nicolas.pitre@linaro.org>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Russell King reports:
| On the ARM dev boards, we have a 32-bit counter running at 24MHz.  Calling
| clocks_calc_mult_shift(&mult, &shift, 24MHz, NSEC_PER_SEC, 60) gives
| us a multiplier of 2796202666 and a shift of 26.
|
| Over a large counter delta, this produces an error - lets take a count
| from 362976315 to 4280663372:
|
| (4280663372-362976315) * 2796202666 / 2^26 - (4280663372-362976315) * (1000/24)
|  => -38.91872422891230269990
|
| Can we do better?
|
| (4280663372-362976315) * 2796202667 / 2^26 - (4280663372-362976315) * (1000/24)
| 19.45936211449532822051
|
| which is about twice as good as the 2796202666 multiplier.
|
| Looking at the equivalent divisions obtained, 2796202666 / 2^26 gives
| 41.66666665673255920410ns per tick, whereas 2796202667 / 2^26 gives
| 41.66666667163372039794ns.  The actual value wanted is 1000/24 =
| 41.66666666666666666666ns.

Fix this by ensuring we round to nearest when calculating the
multiplier.
Signed-off-by: NJohn Stultz <john.stultz@linaro.org>
Tested-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Tested-by: NWill Deacon <will.deacon@arm.com>
Tested-by: NMikael Pettersson <mikpe@it.uu.se>
Tested-by: NEric Miao <eric.y.miao@gmail.com>
Tested-by: NOlof Johansson <olof@lixom.net>
Tested-by: NJamie Iles <jamie@jamieiles.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

ftrace requires sched_clock() to be notrace.  Ensure that all
implementations are so marked.  Also make sure that they include
linux/sched.h

Also ensure OMAP clocksource read functions are marked notrace as
they're used for sched_clock() too.
Tested-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Tested-by: NWill Deacon <will.deacon@arm.com>
Tested-by: NMikael Pettersson <mikpe@it.uu.se>
Tested-by: NEric Miao <eric.y.miao@gmail.com>
Tested-by: NOlof Johansson <olof@lixom.net>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

tegra_clocksource_read should not use cnt32_to_63, wrapping is
already handled in the clocksource code.  Move the cnt32_to_63
into the sched_clock function, and replace the use of clocksource
mult and shift with a multiplication by 1000 to convert us to ns.
Acked-by: NJohn Stultz <johnstul@us.ibm.com>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Tested-by: NOlof Johansson <olof@lixom.net>
Signed-off-by: NColin Cross <ccross@android.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Tested-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Acked-by: NNicolas Pitre <nico@fluxnic.net>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Acked-by: NViresh Kumar <viresh.kumar@st.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Acked-by: NSascha Hauer <s.hauer@pengutronix.de>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Acked-by: NWan zongshun <mcuos.com@gmail.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Acked-by: NKukjin Kim <kgene.kim@samsung.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Tested-by: NEric Miao <eric.y.miao@gmail.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Tested-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Tested-By: NJeff Ohlstein <johlstei@codeaurora.org>
Acked-by: NDavid Brown <davidb@codeaurora.org>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Tested-by: NMikael Pettersson <mikpe@it.uu.se>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Acked-by: NKevin Hilman <khilman@deeprootsystems.com>
Tested-by: NKevin Hilman <khilman@deeprootsystems.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Acked-By: NScott Branden <sbranden@broadcom.com>
Acked-By: NJiandong Zheng <jdzheng@broadcom.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

In d7e81c26 (clocksource: Add clocksource_register_hz/khz interface) new
interfaces were added which simplify (and optimize) the selection of the
divisor shift/mult constants.  Switch over to using this new interface.
Acked-by: NJean-Christophe PLAGNIOL-VILLARD <plagnioj@jcrosoft.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>