The only real need for this field was in
i915_{request,release}_power_well, but there we can get at it by a
container_of magic. Also since in the future we'll have multiple power
wells each with its own power_well struct it makes sense to remove the
field from there where it'd be just redundancy.
Suggested-by: NPaulo Zanoni <paulo.zanoni@intel.com>
Signed-off-by: NImre Deak <imre.deak@intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Currently we make sure that all power domains are enabled during driver
init and turn off unneded ones only after the first modeset. Similarly
during suspend we enable all power domains, which will remain on through
the following resume until the first modeset.

This logic is supported by intel_set_power_well() in the power domain
framework. It would be nice to simplify the API, so that we only have
get/put functions and make it more explicit on the higher level how this
"power well on during init" logic works. This will make it also easier
if in the future we want to shorten the time the power wells are on.

For this add a new device private flag tracking whether we have the
power wells on because of init/suspend and use only
intel_display_power_get()/put(). As nothing else uses
intel_set_power_well() we can remove it.

This also fixes

commit 6efdf354
Author: Imre Deak <imre.deak@intel.com>
Date:   Wed Oct 16 17:25:52 2013 +0300

    drm/i915: enable only the needed power domains during modeset

where removing intel_set_power_well() resulted in not releasing the
reference on the power well that was taken during init and thus leaving
the power well on all the time. Regression reported by Paulo.

v2:
- move the init_power_on flag to the power_domains struct (Daniel)

v3:
- add note about this being a regression fix too (Paulo)
Signed-off-by: NImre Deak <imre.deak@intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

In the future we'll need to support multiple power wells, so prepare for
that here. Create a new power domains struct which contains all
power domain/well specific fields. Since we'll have one lock protecting
all power wells, move power_well->lock to the new struct too.

No functional change.
Signed-off-by: NImre Deak <imre.deak@intel.com>
Reviewed-by: NPaulo Zanoni <paulo.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Production HSW does not need it. I confirmed this with Art.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Production IVB does not need it. I confirmed this with Art.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

All our registers which are written through the MCHBAR are defined
descriptively as an offset to the MCHBAR. We had 3 outliers here.
Convert these as well so all registers which are offsets are MCHBAR can
be easily identified/found within the code.

With this, convert DCLK to also follow this format.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NJani Nikula <jani.nikula@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

There is no hard need for this to be a spin lock, as we don't take these
locks in irq context from anywhere. An upcoming patch will add calls to
punit read/write functions from within regions protected by this lock
and those functions need a mutex in turn. As a solution for that convert
the spin lock to be a mutex.
Signed-off-by: NImre Deak <imre.deak@intel.com>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

It is just cleaner this way and makes it easier to add support for
other HW generations with always-on power wells powering a different
set of domains.
Signed-off-by: NImre Deak <imre.deak@intel.com>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Since we use intel_enable_rc6() now for more than just when we're
enabling RC6, we'll see this message many times, and it is just
confusing.

As an example, calc_residency calls this function whenever poked via
sysfs. This leaves the impression in dmesg that we're constantly
re-enabling RC6.

While at it, move the defines and description from drv.h to intel_pm.c,
since these are only ever used in that code.
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Using the 5/6 DDB split make sense only when sprites are enabled.
So check that before we waste any cycles computing the merged
watermarks with the 5/6 DDB split.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Makes the behaviour of the function more clear.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Makes the intention more clear.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

This debug print just adds overhead to the watermark merging process,
and doesn't really give enough information to be useful. Just kill
and let's add something much better a bit later.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Fill out the HSW watermark s/w tracking structures with the current
hardware state in intel_modeset_setup_hw_state(). This allows us to skip
the HW state readback during watermark programming and just use the values
we keep around in dev_priv->wm. Reduces the overhead of the watermark
programming quite a bit.

v2: s/init_wm/wm_get_hw_state
    Remove stale comment about sprites
    Make DDB partitioning readout safer
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
[danvet: Fix whitespace fail.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Currently hsw_write_vm_values() may write to certain watermark
registers needlessly. For instance if only, say, LP3 changes,
the current code will again disable all LP1+ watermarks even
though only LP3 needs to be reconfigured.

Add an easy to read function that will compute the dirtyness of the
watermarks, and use that information to further optimize the watermark
programming.

v2: Disable LP1+ watermarks around changing LP0 watermarks for Paulo
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

To make it easier to check what watermark updates are actually
necessary, keep copies of the relevant bits that match the current
hardware state.

Also add DDB partitioning into hsw_wm_values as that's another piece
of state we want to track.

We don't read out the hardware state on init yet, so we can't really
start using this yet, but it will be used later.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
[danvet: Paulo asked for a comment around the memcmp to say that we
depend upon zero-initializing the entire structures due to padding.
But a later patch in this series removes the memcmp again. So this is
ok as-is.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The fbc_wm_enabled member in intel_wm_config is useless for the time
being. The original idea for it was that we'd pre-compute it and so
that the WM merging process could know whether it needs to worry
about FBC watermarks at all.

But we don't have a convenient way to pre-check for the possibility
of FBC being used. intel_update_fbc() should be split up for that.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

On HSW the LP1,LP2,LP3 levels are either 1,2,3 or 1,3,4. We make the
conversion from LPn to to the level at one point current. Later we're
going to do it in a few places, so move it to a separate function.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Using the 5/6 DDB split make sense only when sprites are enabled.
So check that before we waste any cycles computing the merged
watermarks with the 5/6 DDB split.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Move the watermark max computations into haswell_update_wm(). This
allows keeping the 1/2 vs. 5/6 split code in one place, and avoid having
to pass around so many things. We also save a bit of stack space by only
requiring one copy of struct hsw_wm_maximums.

Also move the intel_wm_config out from hsw_compute_wm_parameters() and
pass it it. We'll have some need for it in haswell_update_wm() later.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Let's try to keep using the intermediate intel_pipe_wm representation
for as long as possible. It avoids subtle knowledge about the
internals of the hardware registers when trying to choose the
best watermark configuration.

While at it replace the memset() w/ zero initialization.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

I want to convert hsw_find_best_result() to use intel_pipe_wm, so we
need to move the merging to happen outside hsw_compute_wm_results().
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

No point in re-computing the watermarks for all pipes, when only one
pipe has changed. The watermarks stored under intel_crtc.wm.active are
still valid for the other pipes. We just need to redo the merging.

We can also skip the merge/update procedure completely if the new
watermarks for the affected pipe come out unchanged.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Introduce a new struct intel_pipe_wm which contains all the
watermarks for a single pipe. Use it to unify the LP0 and LP1+
watermark computations so that we can just iterate through the
watermark levels neatly and call ilk_compute_wm_level() for each.

Also add another tool ilk_wm_merge() that merges the LP1+ watermarks
from all pipes. For that, embed one intel_pipe_wm inside intel_crtc that
contains the currently valid watermarks for each pipe.

This is mainly preparatory work for pre-computing the watermarks for
each pipe and merging them at a later time. For now the merging still
happens immediately.

v2: Add some comments about level 0 DDB split and intel_wm_config
    Add WARN_ON for level 0 being disabled
    s/lp_wm/merged
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

As we delay the initial RPS enabling (upon boot and after resume), there
is a chance that we may start to render and trigger RPS boosts before we
set up the punit. Any changes we make could result in inconsistent
hardware state, with a danger of causing undefined behaviour. However,
as the boosting is a optional tweak to RPS, we can simply ignore it
whilst RPS is not yet enabled.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Let's try to avoid these confusing negated booleans.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NJani Nikula <jani.nikula@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The policy's max frequency is not equal to the CPU's max frequency. The
ring frequency is derived from the CPU frequency, and not the policy
frequency.

One example of how this may differ through sysfs. If the sysfs max
frequency is modified, that will be used for the max ring frequency
calculation.
(/sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq). As far as I
know, no current governor uses anything but max as the default, but in
theory, they could. Similarly distributions might set policy as part of
their init process.

It's ideal to use the real frequency because when we're currently scaled
up on the GPU. In this case we likely want to race to idle, and using a
less than max ring frequency is non-optimal for this situation.

AFAIK, this patch should have no impact on a majority of people.

This behavior hasn't been changed since it was first introduced:
commit 23b2f8bb
Author: Jesse Barnes <jbarnes@virtuousgeek.org>
Date:   Tue Jun 28 13:04:16 2011 -0700

    drm/i915: load a ring frequency scaling table v3

CC: Jesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NRodrigo Vivi <rodrigo.vivi@gmail.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

The intel_flush_primary_plane name actually tells us which plane
we're talking about.

Also reorganize the internals a bit and add a missing POSTING_READ()
to make sure the hardware has seen the changes by the time we
return from the function.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

drm_vblank_init() is too ugly. Make it a bit easier on the eye by
collecting all the per-crtc vblank counters, timestamps etc. to
a structure and just allocate an array of those.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NDaniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

After applying wait-boost we often find ourselves stuck at higher clocks
than required. The current threshold value requires the GPU to be
continuously and completely idle for 313ms before it is dropped by one
bin. Conversely, we require the GPU to be busy for an average of 90% over
a 84ms period before we upclock. So the current thresholds almost never
downclock the GPU, and respond very slowly to sudden demands for more
power. It is easy to observe that we currently lock into the wrong bin
and both underperform in benchmarks and consume more power than optimal
(just by repeating the task and measuring the different results).

An alternative approach, as discussed in the bspec, is to use a
continuous threshold for upclocking, and an average value for downclocking.
This is good for quickly detecting and reacting to state changes within a
frame, however it fails with the common throttling method of waiting
upon the outstanding frame - at least it is difficult to choose a
threshold that works well at 15,000fps and at 60fps. So continue to use
average busy/idle loads to determine frequency change.

v2: Use 3 power zones to keep frequencies low in steady-state mostly
idle (e.g. scrolling, interactive 2D drawing), and frequencies high
for demanding games. In between those end-states, we use a
fast-reclocking algorithm to converge more quickly on the desired bin.

v3: Bug fixes - make sure we reset adj after switching power zones.

v4: Tune - drop the continuous busy thresholds as it prevents us from
choosing the right frequency for glxgears style swap benchmarks. Instead
the goal is to be able to find the right clocks irrespective of the
wait-boost.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Kenneth Graunke <kenneth@whitecape.org>
Cc: Stéphane Marchesin <stephane.marchesin@gmail.com>
Cc: Owen Taylor <otaylor@redhat.com>
Cc: "Meng, Mengmeng" <mengmeng.meng@intel.com>
Cc: "Zhuang, Lena" <lena.zhuang@intel.com>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

If we encounter a situation where the CPU blocks waiting for results
from the GPU, give the GPU a kick to boost its the frequency.

This should work to reduce user interface stalls and to quickly promote
mesa to high frequencies - but the cost is that our requested frequency
stalls high (as we do not idle for long enough before rc6 to start
reducing frequencies, nor are we aggressive at down clocking an
underused GPU). However, this should be mitigated by rc6 itself powering
off the GPU when idle, and that energy use is dependent upon the workload
of the GPU in addition to its frequency (e.g. the math or sampler
functions only consume power when used). Still, this is likely to
adversely affect light workloads.

In particular, this nearly eliminates the highly noticeable wake-up lag
in animations from idle. For example, expose or workspace transitions.
(However, given the situation where we fail to downclock, our requested
frequency is almost always the maximum, except for Baytrail where we
manually downclock upon idling. This often masks the latency of
upclocking after being idle, so animations are typically smooth - at the
cost of increased power consumption.)

Stéphane raised the concern that this will punish good applications and
reward bad applications - but due to the nature of how mesa performs its
client throttling, I believe all mesa applications will be roughly
equally affected. To address this concern, and to prevent applications
like compositors from permanently boosting the RPS state, we ratelimit the
frequency of the wait-boosts each client recieves.

Unfortunately, this techinique is ineffective with Ironlake - which also
has dynamic render power states and suffers just as dramatically. For
Ironlake, the thermal/power headroom is shared with the CPU through
Intelligent Power Sharing and the intel-ips module. This leaves us with
no GPU boost frequencies available when coming out of idle, and due to
hardware limitations we cannot change the arbitration between the CPU and
GPU quickly enough to be effective.

v2: Limit each client to receiving a single boost for each active period.
    Tested by QA to only marginally increase power, and to demonstrably
    increase throughput in games. No latency measurements yet.

v3: Cater for front-buffer rendering with manual throttling.

v4: Tidy up.

v5: Sadly the compositor needs frequent boosts as it may never idle, but
due to its picking mechanism (using ReadPixels) may require frequent
waits. Those waits, along with the waits for the vrefresh swap, conspire
to keep the GPU at low frequencies despite the interactive latency. To
overcome this we ditch the one-boost-per-active-period and just ratelimit
the number of wait-boosts each client can receive.
Reported-and-tested-by: NPaul Neumann <paul104x@yahoo.de>
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=68716Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Cc: Kenneth Graunke <kenneth@whitecape.org>
Cc: Stéphane Marchesin <stephane.marchesin@gmail.com>
Cc: Owen Taylor <otaylor@redhat.com>
Cc: "Meng, Mengmeng" <mengmeng.meng@intel.com>
Cc: "Zhuang, Lena" <lena.zhuang@intel.com>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
[danvet: No extern for function prototypes in headers.]
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

This patch attempts to clean up the ring/IA scaling programming in the
following ways.
1. Fix the comment about the DDR frequency. The math is 266MHz, not
133MHz. Formula was right, docs are wrong.

2. Mask the DCLK register since I don't know how it is defined on future
platforms.

3. use mult_frac instead of magic math.

This helps for future platform enabling.

v2: Actually use the right patch. The v1 was a mix of things, none of
which was right. Note that due to rounding, we actually get different
values (slightly higher) for the effective ring frequency.

v3: Use 1.25 instead of 1.33 as the original code did. (Jesse)

CC: Jesse Barnes <jbarnes@virtuousgeek.org>
CC: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Reviewed-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Signed-off-by: NDamien Lespiau <damien.lespiau@intel.com>
Reviewed-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

It indicates a probable BIOS bug, but it appears to be harmless, and
there's nothing the user can do about it anyway, so reduce to a debug
msg.  I've filed a bug with the BIOS folks about it anyway, so hopefully
they'll fix whatever GT SB read they were doing when the GT was off.

References: https://bugs.freedesktop.org/show_bug.cgi?id=69396Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

And add some reg defines while we're at it.  Since the units of the RC6
residency counter are actually in CZ clocks, we want to just use the
high bits or we'll overflow too frequently.
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>
Reviewed-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

struct drm_mode_display now has a separate crtc_ version of the clock to
be used when we're talking about the timings given to the harwadre (was
far as the mode is concerned).

This commit is really the result of a git grep adjusted_mode.*clock and
replacing those by adjusted_mode.crtc_clock. No functional change.

v2: Rebased on drm-intel-queued-next
Reviewed-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: NDamien Lespiau <damien.lespiau@intel.com>
Acked-by: NDave Airlie <airlied@gmail.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Done while reviewing all our allocations for fubar. Also a few errant
cases of lacking () for the sizeof operator - just a bit of OCD.

I've left out all the conversions that also should use kcalloc from
this patch  (it's only 2).
Reviewed-by: NJani Nikula <jani.nikula@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

VGA registers/memory live inside the the display power well. Add a power
domain for VGA.
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

We increase/decrease the power well refcount in several places now, and
all of those places need to do the same thing, so pull that code into
a few small helper functions.

v2: Rename the funcs to __intel_power_well_{get,put}
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>

Add APIs to get/put power well references for specific purposes.

v2: Split the i915_request change to another patch
Signed-off-by: NVille Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: NPaulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>