On AGP system we might allocate/free routinely uncached or wc memory,
changing page from cached (wb) to uc or wc is very expensive and involves
a lot of flushing. To improve performance this allocator use a pool
of uc,wc pages.

Pools are protected with spinlocks to allow multiple threads to allocate pages
simultanously. Expensive operations are done outside of spinlock to maximize
concurrency.

Pools are linked lists of pages that were recently freed. mm shrink callback
allows kernel to claim back pages when they are required for something else.

Fixes:
* set_pages_array_wb handles highmem pages so we don't have to remove them
  from pool.
* Add count parameter to ttm_put_pages to avoid looping in free code.
* Change looping from _safe to normal in pool fill error path.
* Initialize sum variable and make the loop prettier in get_num_unused_pages.

* Moved pages_freed reseting inside the loop in ttm_page_pool_free.
* Add warning comment about spinlock context in ttm_page_pool_free.

Based on Jerome Glisse's and Dave Airlie's pool allocator.
Signed-off-by: NJerome Glisse <jglisse@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>
Signed-off-by: NPauli Nieminen <suokkos@gmail.com>
Reviewed-by: NJerome Glisse <jglisse@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Replace sequential calls to kobject_init() and kobject_add() with the
combo wrapper kobject_init_and_add(), which provides the same
semantics.
Signed-off-by: NRobert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Now that the drm core can do this, lets just use it, split the code out
so TTM doesn't have to drag all of drmP.h in.
Signed-off-by: NDave Airlie <airlied@redhat.com>

Constify struct sysfs_ops.

This is part of the ops structure constification
effort started by Arjan van de Ven et al.

Benefits of this constification:

 * prevents modification of data that is shared
   (referenced) by many other structure instances
   at runtime

 * detects/prevents accidental (but not intentional)
   modification attempts on archs that enforce
   read-only kernel data at runtime

 * potentially better optimized code as the compiler
   can assume that the const data cannot be changed

 * the compiler/linker move const data into .rodata
   and therefore exclude them from false sharing
Signed-off-by: NEmese Revfy <re.emese@gmail.com>
Acked-by: NDavid Teigland <teigland@redhat.com>
Acked-by: NMatt Domsch <Matt_Domsch@dell.com>
Acked-by: NMaciej Sosnowski <maciej.sosnowski@intel.com>
Acked-by: NHans J. Koch <hjk@linutronix.de>
Acked-by: NPekka Enberg <penberg@cs.helsinki.fi>
Acked-by: NJens Axboe <jens.axboe@oracle.com>
Acked-by: NStephen Hemminger <shemminger@vyatta.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

- Without this change I get a general protection fault.
- Also use PTR_ERR where applicable.
Signed-off-by: NMaarten Maathuis <madman2003@gmail.com>
Reviewed-by: NDave Airlie <airlied@redhat.com>
Acked-by: NThomas Hellstrom <thellstrom@vmware.com>
Cc: stable@kernel.org
Signed-off-by: NDave Airlie <airlied@redhat.com>

http://bugzilla.kernel.org/show_bug.cgi?id=15328

This fixes a serious regression on AGP/non-PAT systems, where
pages were ending up in the wrong state and slowing down the
whole system.

[airlied: taken this from the bug as the other option is to revert
the change which caused it].

Tested-by: John W. Linville (in bug).
Signed-off-by: NDave Airlie <airlied@redhat.com>

If the buffer object was already in the requested memory type, but
outside of the requested range it was never moved into the requested range.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Signed-off-by: NAustin Yuan <shengquan.yuan@gmail.com>
Acked-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Re-order structure ttm_ref_object to remove 8 bytes of alignment padding
on 64 bit builds, so shrinking its size from 72 to 64 bytes allowing it
to fit into a smaller slab.
Signed-off-by: NRichard Kennedy <richard@rsk.demon.co.uk>
Acked-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Fixes errors like:
> reserve_ram_pages_type failed 0x15b7a000-0x15b7b000, track 0x8, req 0x10
when a BO is moved between WC and UC areas.
Reported-by: NXavier Chantry <shiningxc@gmail.com>
Signed-off-by: NFrancisco Jerez <currojerez@riseup.net>
Acked-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

This is needed to fix a vmwgfx memory usage bug.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Resending this with Thomas Hellstrom's signoff for merging into 2.6.33

ttm_bo_delayed_delete has a race condition, because after we do:
kref_put(&nentry->list_kref, ttm_bo_release_list);

we are not holding the list lock and not holding any reference to
objects, and thus every bo in the list can be removed and freed at
this point.

However, we then use the next pointer we stored, which is not guaranteed
to be valid.

This was apparently the cause of some Nouveau oopses I experienced.

This patch rewrites the function so that it keeps the reference to nentry
until nentry itself is freed and we already got a reference to nentry->next.

v2 updated by me according to Thomas Hellstrom's feedback.
v3 proposed by Thomas Hellstrom. Commit comment updated by me.

Both updates fixed minor efficiency/style issues only and all three versions
should be correct.
Signed-off-by: NLuca Barbieri <luca@luca-barbieri.com>
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

This is a convention that the vmwgfx driver has come to rely on.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

This is needed for a bugfix in the vmwgfx driver.
Drivers may have GPU bindings on buffers that core TTM is not aware of,
and TTM may view those buffers as ordinary system memory buffers.
Add a notifier to such drivers when TTM is about to move the buffer
contents out to swappable memory. The driver must then release any
private GPU bindings on those buffers.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

System memory type doesn't have a drm_mm manager associated to
it. This patch avoid trying to call drm_mm_debug on unitialized
drm_mm when printing debug info on the system memory manager.
Signed-off-by: NJerome Glisse <jglisse@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Signed-off-by: NJerome Glisse <jglisse@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

1) The function was previously called with a potentially empty
LRU list which would have lead to an OOPS or servere corruption.
2) In rare cases, after reservation has succeeded, another process may
already have evicted it or even pinned it. We must revalidate the
buffer status after releasing the lru lock.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

1) Remove from lru before reserving so we avoid competing with
evicting processes.
2) Avoid calling kref_put() on bo::list_kref while spinlocked.
3) Additional refcounting bug-checking.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

a) the loops were going to <= not <, leading to illegal memory access
b) the busy placement checks were using the placement arrays not the
   busy placement ones.
Acked-by: NJerome Glisse <jglisse@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

This path isn't used by radeon yet, but future drivers will want it,
so fix it right.
Reported-by: NLuca Barbieri <luca@luca-barbieri.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

These are functions required by nouveau which will be merged later.
Signed-off-by: NBen Skeggs <bskeggs@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Convert ttm_buffer_object_init to use struct ttm_placement and
rename to ttm_bo_init for consistency with function naming. This
allow to give more complex placement at buffer creation. For
instance you ask to allocate bo into vram first but if there is
not enough vram you can give system as a second possible
placement. It also allow to create buffer in a specific range.

Also rename ttm_buffer_object_validate to ttm_bo_validate.
Signed-off-by: NJerome Glisse <jglisse@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

This add helper function to print information on eviction placements
and memory manager status when eviction fails to allocate memory
space.
Signed-off-by: NJerome Glisse <jglisse@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

This would allow to catch driver callback error of not properly
setting the eviction placement structure.
Signed-off-by: NJerome Glisse <jglisse@redhat.com>
Signed-off-by: NDave Airlie <airlied@linux.ie>

Return -ERESTARTSYS instead of -ERESTART when interrupted by a signal.
The -ERESTARTSYS is converted to an -EINTR by the kernel signal layer
before returned to user-space.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NJerome Glisse <jglisse@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

This change allow driver to pass sorted memory placement,
from most prefered placement to least prefered placement.
In order to avoid long function prototype a structure is
used to gather memory placement informations such as range
restriction (if you need a buffer to be in given range).
Range restriction is determined by fpfn & lpfn which are
the first page and last page number btw which allocation
can happen. If those fields are set to 0 ttm will assume
buffer can be put anywhere in the address space (thus it
avoids putting a burden on the driver to always properly
set those fields).

This patch also factor few functions like evicting first
entry of lru list or getting a memory space. This avoid
code duplication.

V2: Change API to use placement flags and array instead
    of packing placement order into a quadword.
V3: Make sure we set the appropriate mem.placement flag
    when validating or allocation memory space.

[Pending Thomas Hellstrom further review but okay
from preliminary review so far].
Signed-off-by: NJerome Glisse <jglisse@redhat.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

If we don't need the zone we need to free it.
Acked-By: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

None of the in-tree drivers use user objects yet so this wasn't hitting
us.

Stanse found unreachable code in ttm_bo_add_ttm:
http://decibel.fi.muni.cz/~xslaby/stanse/error.cgi?db=32&id=714#l238Reported-by: NJiri Slaby <jirislaby@gmail.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Utilities to reserve, unreserve and fence a list of TTM
buffer objects in a deadlock-safe manner.

Used by the vmwgfx driver.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

This is intended to be used by ttm-aware drivers to
1) Block clients to inactive masters when
they try to validate buffers for GPU use.
2) Optionally block clients to the current master when
there is thrashing due to GPU memory shortage.

Used by the vmwgfx driver.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

Add objects needed for user-space to maintain reference counts on ttm objects.
This is used by the vmwgfx driver which allows user-space to maintain
map-counts on dma buffers, lock-counts on the ttm lock and ref-counts on
gpu surfaces, gpu contexts and dma buffer.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

That is "success", "unknown", "through", "performance", "[re|un]mapping"
, "access", "default", "reasonable", "[con]currently", "temperature"
, "channel", "[un]used", "application", "example","hierarchy", "therefore"
, "[over|under]flow", "contiguous", "threshold", "enough" and others.
Signed-off-by: NAndré Goddard Rosa <andre.goddard@gmail.com>
Signed-off-by: NJiri Kosina <jkosina@suse.cz>

I moved the allocation until after the check for (si->totalhigh == 0).
Signed-off-by: NDan Carpenter <error27@gmail.com>
Acked-By: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>

When we are evicting from VRAM->RAM we allocate the ttm object,
but we don't set the caching policy on it before blitting into it.
This means on AGP we end up blitting into cached pages, and
the CPU later flushes out on top of them. This was mostly seen as
font corruption.

The other question is why we don't evict VRAM->GTT in a lot of cases,
this would save us some cache transitions since a lot of objects
that are evicted from VRAM will probably end up being pulled back in
a few operations later, and evicting them to system memory involves
2 unnecessary cache transitions.
Signed-off-by: NDave Airlie <airlied@redhat.com>

the global refcount wasn't being increased after the first reference.
this caused an oops on unload on a multi-gpu card.
Signed-off-by: NDave Airlie <airlied@redhat.com>

* mark struct vm_area_struct::vm_ops as const
* mark vm_ops in AGP code

But leave TTM code alone, something is fishy there with global vm_ops
being used.
Signed-off-by: NAlexey Dobriyan <adobriyan@gmail.com>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

This merges the TTM and drm cache flushing into one file in the
drm core.
Signed-off-by: NDave Airlie <airlied@redhat.com>

ttm:
Fix error paths when kobject_add returns an error.
Signed-off-by: NThomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: NDave Airlie <airlied@redhat.com>