Removes sparse warnings.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

This is just a variable rename and comment change.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Slightly shorter and easier to read code.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

I found another victim of the radix tree being hard to use.  Because
there was no call to radix_tree_preload(), khugepaged was allocating
radix_tree_nodes using GFP_ATOMIC.

I also converted a local_irq_save()/restore() pair to
disable()/enable().
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Quite a straightforward conversion.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Signed-off-by: NMatthew Wilcox <willy@infradead.org>

This one is trivial.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Both callers of __delete_from_swap_cache have the swp_entry_t already,
so pass that in to make constructing the XA_STATE easier.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Combine __add_to_swap_cache and add_to_swap_cache into one function
since there is no more need to preload.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

This is essentially xa_cmpxchg() with the locking handled above us,
and it doesn't have to handle replacing a NULL entry.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

We construct an XA_STATE and use it to delete the node with
xas_store() rather than adding a special function for this unique
use case.  Includes a test that simulates this usage for the
test suite.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Includes moving mapping_tagged() to fs.h as a static inline, and
changing it to return bool.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Instead of calling find_get_pages_range() and putting any reference,
use xas_find() to iterate over any entries in the range, skipping the
shadow/swap entries.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Rename the function from page_cache_tree_delete_batch to just
page_cache_delete_batch.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Now the page cache lookup is using the XArray, let's convert this
regression test from the radix tree API to the XArray so it's testing
roughly the same thing it was testing before.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Slight change of strategy here; if we have trouble getting hold of a
page for whatever reason (eg a compound page is split underneath us),
don't spin to stabilise the page, just continue the iteration, like we
would if we failed to trylock the page.  Since this is a speculative
optimisation, it feels like we should allow the process to take an extra
fault if it turns out to need this page instead of spending time to pin
down a page it may not need.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Slightly shorter and simpler code.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

The 'end' parameter of the xas_for_each iterator avoids a useless
iteration at the end of the range.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

There's no direct replacement for radix_tree_for_each_contig()
in the XArray API as it's an unusual thing to do.  Instead,
open-code a loop using xas_next().  This removes the only user of
radix_tree_for_each_contig() so delete the iterator from the API and
the test suite code for it.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

The 'end' parameter of the xas_for_each iterator avoids a useless
iteration at the end of the range.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Slightly shorter and simpler code.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Slightly shorter and simpler code.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

The code is slightly shorter and simpler.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Use the XArray APIs to add and replace pages in the page cache.  This
removes two uses of the radix tree preload API and is significantly
shorter code.  It also removes the last user of __radix_tree_create()
outside radix-tree.c itself, so make it static.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

The page cache offers the ability to search for a miss in the previous or
next N locations.  Rather than teach the XArray about the page cache's
definition of a miss, use xas_prev() and xas_next() to search the page
array.  This should be more efficient as it does not have to start the
lookup from the top for each index.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Change i_pages from a radix_tree_root to an xarray, convert the
documentation into kernel-doc format and change the order of the elements
to pack them better on 64-bit systems.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Use the XA_TRACK_FREE ability to track which entries have a free bit,
similarly to how it uses the radix tree's IDR_FREE tag.  This eliminates
the per-cpu ida_bitmap preload, and fixes the memory consumption
regression I introduced when making the IDR able to store any pointer.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Add the optional ability to track which entries in an XArray are free
and provide xa_alloc() to replace most of the functionality of the IDR.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Add myself as XArray and IDR maintainer.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

This function reserves a slot in the XArray for users which need
to acquire multiple locks before storing their entry in the tree and
so cannot use a plain xa_store().
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

This hopefully temporary function is useful for users who have not yet
been converted to multi-index entries.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

This iterator iterates over each entry that is stored in the index or
indices specified by the xa_state.  This is intended for use for a
conditional store of a multiindex entry, or to allow entries which are
about to be removed from the xarray to be disposed of properly.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

The xas_next and xas_prev functions move the xas index by one position,
and adjust the rest of the iterator state to match it.  This is more
efficient than calling xas_set() as it keeps the iterator at the leaves
of the tree instead of walking the iterator from the root each time.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

This function frees all the internal memory allocated to the xarray
and reinitialises it to be empty.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

The xa_extract function combines the functionality of
radix_tree_gang_lookup() and radix_tree_gang_lookup_tagged().
It extracts entries matching the specified filter into a normal array.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

The xa_for_each iterator allows the user to efficiently walk a range
of the array, executing the loop body once for each entry in that
range that matches the filter.  This commit also includes xa_find()
and xa_find_after() which are helper functions for xa_for_each() but
may also be useful in their own right.

In the xas family of functions, we have xas_for_each(), xas_find(),
xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(),
xas_next_tagged() and xas_pause().
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

Like cmpxchg(), xa_cmpxchg will only store to the index if the current
entry matches the old entry.  It returns the current entry, which is
usually more useful than the errno returned by radix_tree_insert().
For the users who really only want the errno, the xa_insert() wrapper
provides a more convenient calling convention.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

xa_store() differs from radix_tree_insert() in that it will overwrite an
existing element in the array rather than returning an error.  This is
the behaviour which most users want, and those that want more complex
behaviour generally want to use the xas family of routines anyway.

For memory allocation, xa_store() will first attempt to request memory
from the slab allocator; if memory is not immediately available, it will
drop the xa_lock and allocate memory, keeping a pointer in the xa_state.
It does not use the per-CPU cache, although those will continue to exist
until all radix tree users are converted to the xarray.

This patch also includes xa_erase() and __xa_erase() for a streamlined
way to store NULL.  Since there is no need to allocate memory in order
to store a NULL in the XArray, we do not need to trouble the user with
deciding what memory allocation flags to use.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>

XArray marks are like the radix tree tags, only slightly more strongly
typed.  They are renamed in order to distinguish them from tagged
pointers.  This commit adds the basic get/set/clear operations.
Signed-off-by: NMatthew Wilcox <willy@infradead.org>