Change all three callers and the worker function try_to_unmap_one().
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>

Convert split_huge_pmd_address() at the same time since it only passes
the folio through, and its two callers already have a folio on hand.
Removes numerous calls to compound_head() and removes an assumption
that a page cannot be larger than a PMD.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>

Both its callers pass a page which was previously on an LRU list,
so were passing a folio by definition.  Use the type system to enforce
that and remove a few calls to compound_head().
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

Convert mlock_page() into mlock_folio() and convert the callers.  Keep
mlock_vma_page() as a wrapper.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>

folio_mkclean() already passes down a head page, so convert it
back to a folio.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

The PG_idle and PG_young bits are ignored if they're set on tail
pages, so ensure we're passing a folio around.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

page_mapped_in_vma() really just wants to walk one page, but as the
code stands, if passed the head page of a compound page, it will
walk every page in the compound page.  Extract pfn/nr_pages/pgoff
from the struct page early, so they can be overridden by
page_mapped_in_vma().
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>

We need to use this function in common code; pull it out of pmd_page().
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>

This is straightforward for everything except nohash64 where we
indirect through pmd_page().  There must be a better way to do this.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>

Whether or not the platform supports PMD sized pages, we need to
provide pmd_pfn() for an upcoming patch.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>

We need to use this function in common code, so define it for
architectures and/or configrations that miss it.  The result of
pmd_pfn() will only be used if TRANSPARENT_HUGEPAGE is enabled,
but a function or macro called pmd_pfn() must be defined, even
on machines with two level page tables.
Signed-off-by: NMike Rapoport <rppt@linux.ibm.com>
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>

Instead of declaring a struct page_vma_mapped_walk directly,
use these helpers to allow us to transition to a PFN approach in the
following patches.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>

This is the folio equivalent of page_to_pgoff().
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

This is a convenience function; split_huge_page_to_list() can take
any page in a folio (and does so on purpose because that page will
be the one which keeps the refcount).  But it's convenient for the
callers to pass the folio instead of the first page in the folio.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

This implements the same algorithm as total_mapcount(), which is
transformed into a wrapper function.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

Adjust documentation to be more clear.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

Saves a few calls to compound_head().
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

These functions are page cache functionality and don't need to be
declared in fs.h.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NMiaohe Lin <linmiaohe@huawei.com>

We can save a function call by combining these two functions, which
are identical except for the return value.  Also move the prototype
to mm/internal.h.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NMiaohe Lin <linmiaohe@huawei.com>

This function has one caller which already has a reference to the
page, so we don't need to use get_page_unless_zero().  Also move the
prototype to mm/internal.h.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NMiaohe Lin <linmiaohe@huawei.com>

Now we can call mapping_evict_folio() instead of invalidate_inode_page()
and save a few calls to compound_head().
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NMiaohe Lin <linmiaohe@huawei.com>

Some of the callers already have the address_space and can avoid calling
folio_mapping() and checking if the folio was already truncated.  Also
add kernel-doc and fix the return type (in case we ever support folios
larger than 4TB).
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NMiaohe Lin <linmiaohe@huawei.com>

Add kernel-doc and return the number of pages removed in order to
get the statistics right in __invalidate_mapping_pages().
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NMiaohe Lin <linmiaohe@huawei.com>

folio_mapped() is expensive because it has to check each page's mapcount
field.  A cheaper check is whether there are any extra references to
the page, other than the one we own, one from the page private data and
the ones held by the page cache.

The call to remove_mapping() will fail in any case if it cannot freeze
the refcount, but failing here avoids cycling the i_pages spinlock.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NMiaohe Lin <linmiaohe@huawei.com>

This saves a number of calls to compound_head().
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NMiaohe Lin <linmiaohe@huawei.com>

invalidate_inode_page() is the only caller of invalidate_complete_page()
and inlining it reveals that the first check is unnecessary (because we
hold the page locked, and we just retrieved the mapping from the page).
Actually, it does make a difference, in that tail pages no longer fail
at this check, so it's now possible to remove a tail page from a mapping.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NJohn Hubbard <jhubbard@nvidia.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

This saves a lot of calls to compound_head().
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NMiaohe Lin <linmiaohe@huawei.com>

This removes a few hidden calls to compound_head().
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

Add a putback_lru_page() wrapper.  Removes a couple of compound_head()
calls.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

Since page->lru occupies the same bytes as compound_head, any page
on the LRU list must be a folio.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>

This removes an assumption that THPs are the only kind of compound
pages and removes a couple of hidden calls to compound_head.  It
also documents that you can't pass a tail page to mem_cgroup_swapout().
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

This removes an assumption that THPs are the only kind of compound
pages and removes a few hidden calls to compound_head().
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>

Switch from head pages to folios.  This removes an assumption that
THPs are the only way to have a high-order page.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohn Hubbard <jhubbard@nvidia.com>
Reviewed-by: NJason Gunthorpe <jgg@nvidia.com>
Reviewed-by: NWilliam Kucharski <william.kucharski@oracle.com>

folio_is_zone_device() is equivalent to is_zone_device_page(),
folio_is_device_private() is equivalent to is_device_private_page(),
and folio_is_pinnable() is equivalent to is_pinnable_page().

All of these tests return the same result for every page in the folio,
so we can just pass the head page of the folio to the page variant of
the function.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>

Add isolate_lru_page() as a wrapper around isolate_lru_folio().
TestClearPageLRU() would have always failed on a tail page, so
returning -EBUSY is the same behaviour.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohn Hubbard <jhubbard@nvidia.com>
Reviewed-by: NJason Gunthorpe <jgg@nvidia.com>
Reviewed-by: NWilliam Kucharski <william.kucharski@oracle.com>

Convert the only caller to work on folios instead of pages.
This removes the last caller of put_compound_head(), so delete it.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohn Hubbard <jhubbard@nvidia.com>
Reviewed-by: NJason Gunthorpe <jgg@nvidia.com>
Reviewed-by: NWilliam Kucharski <william.kucharski@oracle.com>

Convert both callers to work on folios instead of pages.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohn Hubbard <jhubbard@nvidia.com>
Reviewed-by: NJason Gunthorpe <jgg@nvidia.com>
Reviewed-by: NWilliam Kucharski <william.kucharski@oracle.com>

Use the new folio-based APIs.  This was the last user of
try_grab_compound_head(), so remove it.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohn Hubbard <jhubbard@nvidia.com>
Reviewed-by: NJason Gunthorpe <jgg@nvidia.com>
Reviewed-by: NWilliam Kucharski <william.kucharski@oracle.com>

Use the new folio-based APIs.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohn Hubbard <jhubbard@nvidia.com>
Reviewed-by: NJason Gunthorpe <jgg@nvidia.com>
Reviewed-by: NWilliam Kucharski <william.kucharski@oracle.com>

Use the new folio-based APIs.
Signed-off-by: NMatthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NJohn Hubbard <jhubbard@nvidia.com>
Reviewed-by: NJason Gunthorpe <jgg@nvidia.com>
Reviewed-by: NWilliam Kucharski <william.kucharski@oracle.com>