There is a potential race between a thread charging a page and another
thread putting it back to the LRU list:

  charge:                         putback:
  SetPageCgroupUsed               SetPageLRU
  PageLRU && add to memcg LRU     PageCgroupUsed && add to memcg LRU

The order of setting one flag and checking the other is crucial, otherwise
the charge may observe !PageLRU while the putback observes !PageCgroupUsed
and the page is not linked to the memcg LRU at all.

Global memory pressure may fix this by trying to isolate and putback the
page for reclaim, where that putback would link it to the memcg LRU again.
 Without that, the memory cgroup is undeletable due to a charge whose
physical page can not be found and moved out.
Signed-off-by: NJohannes Weiner <jweiner@redhat.com>
Cc: Ying Han <yinghan@google.com>
Acked-by: NKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Cc: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
Cc: Balbir Singh <bsingharora@gmail.com>
Cc: Michal Hocko <mhocko@suse.cz>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Reclaim decides to skip scanning an active list when the corresponding
inactive list is above a certain size in comparison to leave the assumed
working set alone while there are still enough reclaim candidates around.

The memcg implementation of comparing those lists instead reports whether
the whole memcg is low on the requested type of inactive pages,
considering all nodes and zones.

This can lead to an oversized active list not being scanned because of the
state of the other lists in the memcg, as well as an active list being
scanned while its corresponding inactive list has enough pages.

Not only is this wrong, it's also a scalability hazard, because the global
memory state over all nodes and zones has to be gathered for each memcg
and zone scanned.

Make these calculations purely based on the size of the two LRU lists
that are actually affected by the outcome of the decision.
Signed-off-by: NJohannes Weiner <jweiner@redhat.com>
Reviewed-by: NRik van Riel <riel@redhat.com>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Acked-by: NKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Cc: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
Cc: Balbir Singh <bsingharora@gmail.com>
Reviewed-by: NMinchan Kim <minchan.kim@gmail.com>
Reviewed-by: NYing Han <yinghan@google.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

If somebody is touching data too early, it might be easier to diagnose a
problem when dereferencing NULL at mem->info.nodeinfo[node] than trying to
understand why mem_cgroup_per_zone is [un|partly]initialized.
Signed-off-by: NIgor Mammedov <imammedo@redhat.com>
Acked-by: NMichal Hocko <mhocko@suse.cz>
Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Before calling schedule_timeout(), task state should be changed.
Signed-off-by: NKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Acked-by: NMichal Hocko <mhocko@suse.cz>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

The memcg code sometimes uses "struct mem_cgroup *mem" and sometimes uses
"struct mem_cgroup *memcg".  Rename all mem variables to memcg in source
file.
Signed-off-by: NRaghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
Acked-by: NKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Acked-by: NMichal Hocko <mhocko@suse.cz>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

When the cgroup base was allocated with kmalloc, it was necessary to
annotate the variable with kmemleak_not_leak().  But because it has
recently been changed to be allocated with alloc_page() (which skips
kmemleak checks) causes a warning on boot up.

I was triggering this output:

 allocated 8388608 bytes of page_cgroup
 please try 'cgroup_disable=memory' option if you don't want memory cgroups
 kmemleak: Trying to color unknown object at 0xf5840000 as Grey
 Pid: 0, comm: swapper Not tainted 3.0.0-test #12
 Call Trace:
  [<c17e34e6>] ? printk+0x1d/0x1f^M
  [<c10e2941>] paint_ptr+0x4f/0x78
  [<c178ab57>] kmemleak_not_leak+0x58/0x7d
  [<c108ae9f>] ? __rcu_read_unlock+0x9/0x7d
  [<c1cdb462>] kmemleak_init+0x19d/0x1e9
  [<c1cbf771>] start_kernel+0x346/0x3ec
  [<c1cbf1b4>] ? loglevel+0x18/0x18
  [<c1cbf0aa>] i386_start_kernel+0xaa/0xb0

After a bit of debugging I tracked the object 0xf840000 (and others) down
to the cgroup code.  The change from allocating base with kmalloc to
alloc_page() has the base not calling kmemleak_alloc() which adds the
pointer to the object_tree_root, but kmemleak_not_leak() adds it to the
crt_early_log[] table.  On kmemleak_init(), the entry is found in the
early_log[] but not the object_tree_root, and this error message is
displayed.

If alloc_page() fails then it defaults back to vmalloc() which still uses
the kmemleak_alloc() which makes us still need the kmemleak_not_leak()
call.  The solution is to call the kmemleak_alloc() directly if the
alloc_page() succeeds.
Reviewed-by: NMichal Hocko <mhocko@suse.cz>
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NJonathan Nieder <jrnieder@gmail.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

If a task has exited to the point it has called cgroup_exit() already,
then we can't migrate it to another cgroup anymore.

This can happen when we are attaching a task to a new cgroup between the
call to ->can_attach_task() on subsystems and the migration that is
eventually tried in cgroup_task_migrate().

In this case cgroup_task_migrate() returns -ESRCH and we don't want to
attach the task to the subsystems because the attachment to the new cgroup
itself failed.

Fix this by only calling ->attach_task() on the subsystems if the cgroup
migration succeeded.
Reported-by: NOleg Nesterov <oleg@redhat.com>
Signed-off-by: NBen Blum <bblum@andrew.cmu.edu>
Acked-by: NPaul Menage <paul@paulmenage.org>
Cc: Li Zefan <lizf@cn.fujitsu.com>
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: NFrederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Fix unstable tasklist locking in cgroup_attach_proc.

According to this thread - https://lkml.org/lkml/2011/7/27/243 - RCU is
not sufficient to guarantee the tasklist is stable w.r.t.  de_thread and
exit.  Taking tasklist_lock for reading, instead of rcu_read_lock, ensures
proper exclusion.
Signed-off-by: NBen Blum <bblum@andrew.cmu.edu>
Acked-by: NPaul Menage <paul@paulmenage.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: Neil Brown <neilb@suse.de>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Clement Lecigne reports a filesystem which causes a kernel oops in
hfs_find_init() trying to dereference sb->ext_tree which is NULL.

This proves to be because the filesystem has a corrupted MDB extent
record, where the extents file does not fit into the first three extents
in the file record (the first blocks).

In hfs_get_block() when looking up the blocks for the extent file
(HFS_EXT_CNID), it fails the first blocks special case, and falls
through to the extent code (which ultimately calls hfs_find_init())
which is in the process of being initialised.

Hfs avoids this scenario by always having the extents b-tree fitting
into the first blocks (the extents B-tree can't have overflow extents).

The fix is to check at mount time that the B-tree fits into first
blocks, i.e.  fail if HFS_I(inode)->alloc_blocks >=
HFS_I(inode)->first_blocks

Note, the existing commit 47f365eb ("hfs: fix oops on mount with
corrupted btree extent records") becomes subsumed into this as a special
case, but only for the extents B-tree (HFS_EXT_CNID), it is perfectly
acceptable for the catalog B-Tree file to grow beyond three extents,
with the remaining extent descriptors in the extents overfow.

This fixes CVE-2011-2203
Reported-by: NClement LECIGNE <clement.lecigne@netasq.com>
Signed-off-by: NPhillip Lougher <plougher@redhat.com>
Cc: Jeff Mahoney <jeffm@suse.com>
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Use mpage_readpages() instead of multiple calls to isofs_readpage() to
reduce the CPU utilization and make performance higher.
Signed-off-by: NNamjae Jeon <linkinjeon@gmail.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Jan Kara <jack@suse.cz>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

One can get this information from minix/inode.c, but adding the
explanations at the definition sites is more appropriate.
Signed-off-by: NSami Kerola <kerolasa@iki.fi>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

The driver is added using platform_driver_probe(), so the callbacks can be
discarded more aggessively.
Signed-off-by: NUwe Kleine-König <u.kleine-koenig@pengutronix.de>
Cc: Alessandro Zummo <a.zummo@towertech.it>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Add initial support for the microchip mcp7941x series of real time clocks.

The mcp7941x series is generally compatible with the ds1307 and ds1337 rtc
devices from dallas semiconductor.  minor differences include a backup
battery enable bit, and the polarity of the oscillator enable bit.
Signed-off-by: NDavid Anders <danders.dev@gmail.com>
Cc: Alessandro Zummo <a.zummo@towertech.it>
Reviewed-by: NWolfram Sang <w.sang@pengutronix.de>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

This is the one use of an ida that doesn't retry on receiving -EAGAIN.
I'm assuming do so will cause no harm and may help on a rare occasion.
Signed-off-by: NJonathan Cameron <jic23@cam.ac.uk>
Cc: Alessandro Zummo <a.zummo@towertech.it>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

When a cramfs ramdisk padded with 512 bytes is given to the kernel, the
current identify_ramdisk_image function fails to identify it.

Tested with a padded cramfs image on an ARM based board.
Signed-off-by: NNeil Armstrong <narmstrong@neotion.com>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Davidlohr Bueso <dave@gnu.org>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Since ramfs is hard-selected to "y", the module leftovers make no sense.
Signed-off-by: NRichard Weinberger <richard@nod.at>
Reviewed-by: NWANG Cong <xiyou.wangcong@gmail.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

The case of address space randomization being disabled in runtime through
randomize_va_space sysctl is not treated properly in load_elf_binary(),
resulting in SIGKILL coming at exec() time for certain PIE-linked binaries
in case the randomization has been disabled at runtime prior to calling
exec().

Handle the randomize_va_space == 0 case the same way as if we were not
supporting .text randomization at all.

Based on original patch by H.J. Lu and Josh Boyer.
Signed-off-by: NJiri Kosina <jkosina@suse.cz>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Russell King <rmk@arm.linux.org.uk>
Cc: H.J. Lu <hongjiu.lu@intel.com>
Cc: <stable@kernel.org>
Tested-by: NJosh Boyer <jwboyer@redhat.com>
Acked-by: NNicolas Pitre <nicolas.pitre@linaro.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

This avoids duplicating the function in every arch gup_fast.
Signed-off-by: NAndrea Arcangeli <aarcange@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Hugh Dickins <hughd@google.com>
Cc: Johannes Weiner <jweiner@redhat.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: David Gibson <david@gibson.dropbear.id.au>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: David Miller <davem@davemloft.net>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Up to this point the code assumed old refcounting for hugepages (pre-thp).
 This updates the code directly to the thp mapcount tail page refcounting.
Signed-off-by: NAndrea Arcangeli <aarcange@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Hugh Dickins <hughd@google.com>
Cc: Johannes Weiner <jweiner@redhat.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: David Gibson <david@gibson.dropbear.id.au>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Acked-by: NDavid Miller <davem@davemloft.net>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

s390 didn't return 0 in that case, if it's rolling back the *nr pointer it
should also return zero to avoid adding pages to the array at the wrong
offset.
Signed-off-by: NAndrea Arcangeli <aarcange@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Hugh Dickins <hughd@google.com>
Cc: Johannes Weiner <jweiner@redhat.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: David Gibson <david@gibson.dropbear.id.au>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: David Miller <davem@davemloft.net>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Up to this point the code assumed old refcounting for hugepages (pre-thp).
This updates the code directly to the thp mapcount tail page refcounting.
Signed-off-by: NAndrea Arcangeli <aarcange@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Hugh Dickins <hughd@google.com>
Cc: Johannes Weiner <jweiner@redhat.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: David Gibson <david@gibson.dropbear.id.au>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: David Miller <davem@davemloft.net>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

powerpc didn't return 0 in that case, if it's rolling back the *nr pointer
it should also return zero to avoid adding pages to the array at the wrong
offset.
Signed-off-by: NAndrea Arcangeli <aarcange@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Hugh Dickins <hughd@google.com>
Cc: Johannes Weiner <jweiner@redhat.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Acked-by: NDavid Gibson <david@gibson.dropbear.id.au>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: David Miller <davem@davemloft.net>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Up to this point the code assumed old refcounting for hugepages (pre-thp).
This updates the code directly to the thp mapcount tail page refcounting.
Signed-off-by: NAndrea Arcangeli <aarcange@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Hugh Dickins <hughd@google.com>
Cc: Johannes Weiner <jweiner@redhat.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

We only taken "refs" pins on the head page not "*nr" pins.
Signed-off-by: NAndrea Arcangeli <aarcange@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Hugh Dickins <hughd@google.com>
Cc: Johannes Weiner <jweiner@redhat.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Acked-by: NDavid Gibson <david@gibson.dropbear.id.au>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

"page" may have changed to point to the next hugepage after the loop
completed, The references have been taken on the head page, so the
put_page must happen there too.

This is a longstanding issue pre-thp inclusion.

It's totally unclear how these page_cache_add_speculative and
pte_val(pte) != pte_val(*ptep) checks are necessary across all the
powerpc gup_fast code, when x86 doesn't need any of that: there's no way
the page can be freed with irq disabled so we're guaranteed the
atomic_inc will happen on a page with page_count > 0 (so not needing the
speculative check).

The pte check is also meaningless on x86: no need to rollback on x86 if
the pte changed, because the pte can still change a CPU tick after the
check succeeded and it won't be rolled back in that case.  The important
thing is we got a reference on a valid page that was mapped there a CPU
tick ago.  So not knowing the soft tlb refill code of ppc64 in great
detail I'm not removing the "speculative" page_count increase and the
pte checks across all the code, but unless there's a strong reason for
it they should be later cleaned up too.

If a pte can change from huge to non-huge (like it could happen with
THP) passing a pte_t *ptep to gup_hugepte() would also require to repeat
the is_hugepd in gup_hugepte(), but that shouldn't happen with hugetlbfs
only so I'm not altering that.
Signed-off-by: NAndrea Arcangeli <aarcange@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Hugh Dickins <hughd@google.com>
Cc: Johannes Weiner <jweiner@redhat.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Acked-by: NDavid Gibson <david@gibson.dropbear.id.au>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

This part of gup_fast doesn't seem capable of handling hugetlbfs ptes,
those should be handled by gup_hugepd only, so these checks are
superfluous.

Plus if this wasn't a noop, it would have oopsed because, the insistence
of using the speculative refcounting would trigger a VM_BUG_ON if a tail
page was encountered in the page_cache_get_speculative().
Signed-off-by: NAndrea Arcangeli <aarcange@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Hugh Dickins <hughd@google.com>
Cc: Johannes Weiner <jweiner@redhat.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Acked-by: NDavid Gibson <david@gibson.dropbear.id.au>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Michel while working on the working set estimation code, noticed that
calling get_page_unless_zero() on a random pfn_to_page(random_pfn)
wasn't safe, if the pfn ended up being a tail page of a transparent
hugepage under splitting by __split_huge_page_refcount().

He then found the problem could also theoretically materialize with
page_cache_get_speculative() during the speculative radix tree lookups
that uses get_page_unless_zero() in SMP if the radix tree page is freed
and reallocated and get_user_pages is called on it before
page_cache_get_speculative has a chance to call get_page_unless_zero().

So the best way to fix the problem is to keep page_tail->_count zero at
all times.  This will guarantee that get_page_unless_zero() can never
succeed on any tail page.  page_tail->_mapcount is guaranteed zero and
is unused for all tail pages of a compound page, so we can simply
account the tail page references there and transfer them to
tail_page->_count in __split_huge_page_refcount() (in addition to the
head_page->_mapcount).

While debugging this s/_count/_mapcount/ change I also noticed get_page is
called by direct-io.c on pages returned by get_user_pages.  That wasn't
entirely safe because the two atomic_inc in get_page weren't atomic.  As
opposed to other get_user_page users like secondary-MMU page fault to
establish the shadow pagetables would never call any superflous get_page
after get_user_page returns.  It's safer to make get_page universally safe
for tail pages and to use get_page_foll() within follow_page (inside
get_user_pages()).  get_page_foll() is safe to do the refcounting for tail
pages without taking any locks because it is run within PT lock protected
critical sections (PT lock for pte and page_table_lock for
pmd_trans_huge).

The standard get_page() as invoked by direct-io instead will now take
the compound_lock but still only for tail pages.  The direct-io paths
are usually I/O bound and the compound_lock is per THP so very
finegrined, so there's no risk of scalability issues with it.  A simple
direct-io benchmarks with all lockdep prove locking and spinlock
debugging infrastructure enabled shows identical performance and no
overhead.  So it's worth it.  Ideally direct-io should stop calling
get_page() on pages returned by get_user_pages().  The spinlock in
get_page() is already optimized away for no-THP builds but doing
get_page() on tail pages returned by GUP is generally a rare operation
and usually only run in I/O paths.

This new refcounting on page_tail->_mapcount in addition to avoiding new
RCU critical sections will also allow the working set estimation code to
work without any further complexity associated to the tail page
refcounting with THP.
Signed-off-by: NAndrea Arcangeli <aarcange@redhat.com>
Reported-by: NMichel Lespinasse <walken@google.com>
Reviewed-by: NMichel Lespinasse <walken@google.com>
Reviewed-by: NMinchan Kim <minchan.kim@gmail.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Hugh Dickins <hughd@google.com>
Cc: Johannes Weiner <jweiner@redhat.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: David Gibson <david@gibson.dropbear.id.au>
Cc: <stable@kernel.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

* 'next/soc' of git://git.linaro.org/people/arnd/arm-soc: (21 commits)
  MAINTAINERS: add ARM/FREESCALE IMX6 entry
  arm/imx: merge i.MX3 and i.MX6
  arm/imx6q: add suspend/resume support
  arm/imx6q: add device tree machine support
  arm/imx6q: add smp and cpu hotplug support
  arm/imx6q: add core drivers clock, gpc, mmdc and src
  arm/imx: add gic_handle_irq function
  arm/imx6q: add core definitions and low-level debug uart
  arm/imx6q: add device tree source
  ARM: highbank: add suspend support
  ARM: highbank: Add cpu hotplug support
  ARM: highbank: add SMP support
  MAINTAINERS: add Calxeda Highbank ARM platform
  ARM: add Highbank core platform support
  ARM: highbank: add devicetree source
  ARM: l2x0: add empty l2x0_of_init
  picoxcell: add a definition of VMALLOC_END
  picoxcell: remove custom ioremap implementation
  picoxcell: add the DTS for the PC7302 board
  picoxcell: add the DTS for pc3x2 and pc3x3 devices
  ...

Fix up trivial conflicts in arch/arm/Kconfig, and some more header file
conflicts in arch/arm/mach-omap2/board-generic.c (as per an ealier merge
by Arnd).

* 'next/dt' of git://git.linaro.org/people/arnd/arm-soc:
  ARM: gic: use module.h instead of export.h
  ARM: gic: fix irq_alloc_descs handling for sparse irq
  ARM: gic: add OF based initialization
  ARM: gic: add irq_domain support
  irq: support domains with non-zero hwirq base
  of/irq: introduce of_irq_init
  ARM: at91: add at91sam9g20 and Calao USB A9G20 DT support
  ARM: at91: dt: at91sam9g45 family and board device tree files
  arm/mx5: add device tree support for imx51 babbage
  arm/mx5: add device tree support for imx53 boards
  ARM: msm: Add devicetree support for msm8660-surf
  msm_serial: Add devicetree support
  msm_serial: Use relative resources for iomem

Fix up conflicts in arch/arm/mach-at91/{at91sam9260.c,at91sam9g45.c}

* 'next/cleanup2' of git://git.linaro.org/people/arnd/arm-soc: (31 commits)
  ARM: OMAP: Warn if omap_ioremap is called before SoC detection
  ARM: OMAP: Move set_globals initialization to happen in init_early
  ARM: OMAP: Map SRAM later on with ioremap_exec()
  ARM: OMAP: Remove calls to SRAM allocations for framebuffer
  ARM: OMAP: Avoid cpu_is_omapxxxx usage until map_io is done
  ARM: OMAP1: Use generic map_io, init_early and init_irq
  arm/dts: OMAP3+: Add mpu, dsp and iva nodes
  arm/dts: OMAP4: Add a main ocp entry bound to l3-noc driver
  ARM: OMAP2+: l3-noc: Add support for device-tree
  ARM: OMAP2+: board-generic: Add i2c static init
  ARM: OMAP2+: board-generic: Add DT support to generic board
  arm/dts: Add support for OMAP3 Beagle board
  arm/dts: Add initial device tree support for OMAP3 SoC
  arm/dts: Add support for OMAP4 SDP board
  arm/dts: Add support for OMAP4 PandaBoard
  arm/dts: Add initial device tree support for OMAP4 SoC
  ARM: OMAP: omap_device: Add a method to build an omap_device from a DT node
  ARM: OMAP: omap_device: Add omap_device_[alloc|delete] for DT integration
  of: Add helpers to get one string in multiple strings property
  ARM: OMAP2+: devices: Remove all omap_device_pm_latency structures
  ...

Fix up trivial header file conflicts in arch/arm/mach-omap2/board-generic.c

* 'next/cross-platform' of git://git.linaro.org/people/arnd/arm-soc:
  arm/imx: use Kconfig choice for low-level debug UART selection
  ARM: realview: use Kconfig choice for debug UART selection
  ARM: plat-samsung: use Kconfig choice for debug UART selection
  ARM: versatile: convert logical CPU numbers to physical numbers
  ARM: ux500: convert logical CPU numbers to physical numbers
  ARM: shmobile: convert logical CPU numbers to physical numbers
  ARM: msm: convert logical CPU numbers to physical numbers
  ARM: exynos4: convert logical CPU numbers to physical numbers

Fix up trivial conflict (config DEBUG_S3C_UART move/split vs addition of
ARM_KPROBES_TEST option) in arch/arm/Kconfig.debug

* 'next/devel' of git://git.linaro.org/people/arnd/arm-soc: (50 commits)
  ARM: tegra: update defconfig
  arm/tegra: Harmony: Configure PMC for low-level interrupts
  arm/tegra: device tree support for ventana board
  arm/tegra: add support for ventana pinmuxing
  arm/tegra: prepare Seaboard pinmux code for derived boards
  arm/tegra: pinmux: ioremap registers
  gpio/tegra: Convert to a platform device
  arm/tegra: Convert pinmux driver to a platform device
  arm/dt: Tegra: Add pinmux node to tegra20.dtsi
  arm/tegra: Prep boards for gpio/pinmux conversion to pdevs
  ARM: mx5: fix clock usage for suspend
  ARM i.MX entry-macro.S: remove now unused code
  ARM i.MX boards: use CONFIG_MULTI_IRQ_HANDLER
  ARM i.MX tzic: add handle_irq function
  ARM i.MX avic: add handle_irq function
  ARM: mx25: Add the missing IIM base definition
  ARM i.MX avic: convert to use generic irq chip
  mx31moboard: Add poweroff support
  ARM: mach-qong: Add watchdog support
  ARM: davinci: AM18x: Add wl1271/wlan support
  ...

Fix up conflicts in:
	arch/arm/mach-at91/at91sam9g45.c
	arch/arm/mach-mx5/devices-imx53.h
	arch/arm/plat-mxc/include/mach/memory.h

* 'next/board' of git://git.linaro.org/people/arnd/arm-soc: (34 commits)
  ep93xx: add support Vision EP9307 SoM
  ARM: mxs: Add initial support for DENX MX28
  ARM: EXYNOS4: Add support SMDK4412 Board
  ARM: EXYNOS4: Add MCT support for EXYNOS4412
  ARM: EXYNOS4: Add functions for gic interrupt handling
  ARM: EXYNOS4: Add support clock for EXYNOS4412
  ARM: EXYNOS4: Add support new EXYNOS4412 SoC
  ARM: EXYNOS4: Add support MCT PPI for EXYNOS4212
  ARM: EXYNOS4: Add support PPI in external GIC
  ARM: EXYNOS4: convert boot_params to atag_offset
  ixp4xx: support omicron ixp425 based boards
  ARM: EXYNOS4: Add support SMDK4212 Board
  ARM: EXYNOS4: Add support PM for EXYNOS4212
  ARM: EXYNOS4: Add support clock for EXYNOS4212
  ARM: EXYNOS4: Add support new EXYNOS4212 SoC
  at91: USB-A9G20 C01 & C11 board support
  at91: merge board USB-A9260 and USB-A9263 together
  at91: add support for RSIs EWS board
  ARM: SAMSUNG: Fix mask value for S5P64X0 CPU IDs
  ARM: SAMSUNG: Fix mask for S3C64xx CPU IDs
  ...

* 'next/deletion' of git://git.linaro.org/people/arnd/arm-soc:
  ARM: mach-nuc93x: delete

Fix up trivial delete/edit conflicts in
	arch/arm/mach-nuc93x/{Makefile.boot,mach-nuc932evb.c,time.c}

* 'next/pm' of git://git.linaro.org/people/arnd/arm-soc: (66 commits)
  ARM: CSR: PM: use outer_resume to resume L2 cache
  ARM: CSR: call l2x0_of_init to init L2 cache of SiRFprimaII
  ARM: OMAP: voltage: voltage layer present, even when CONFIG_PM=n
  ARM: CSR: PM: add sleep entry for SiRFprimaII
  ARM: CSR: PM: save/restore irq status in suspend cycle
  ARM: CSR: PM: save/restore timer status in suspend cycle
  OMAP4: PM: TWL6030: add cmd register
  OMAP4: PM: TWL6030: fix ON/RET/OFF voltages
  OMAP4: PM: TWL6030: address 0V conversions
  OMAP4: PM: TWL6030: fix uv to voltage for >0x39
  OMAP4: PM: TWL6030: fix voltage conversion formula
  omap: voltage: add a stub header file for external/regulator use
  OMAP2+: VC: more registers are per-channel starting with OMAP5
  OMAP3+: voltage: update nominal voltage in voltdm_scale() not VC post-scale
  OMAP3+: voltage: rename omap_voltage_get_nom_volt -> voltdm_get_voltage
  OMAP3+: voltdm: final removal of omap_vdd_info
  OMAP3+: voltage: move/rename curr_volt from vdd_info into struct voltagedomain
  OMAP3+: voltage: rename scale and reset functions using voltdm_ prefix
  OMAP3+: VP: combine setting init voltage into common function
  OMAP3+: VP: remove unused omap_vp_get_curr_volt()
  ...

Fix up trivial conflict in arch/arm/mach-prima2/l2x0.c (code removal vs
edit)

* 'next/timer' of git://git.linaro.org/people/arnd/arm-soc:
  clocksource: fixup ux500 build problems
  ARM: omap: use __devexit_p in dmtimer driver
  ARM: ux500: Reprogram timers upon resume
  ARM: plat-nomadik: timer: Export reset functions
  ARM: plat-nomadik: timer: Add support for periodic timers
  ARM: ux500: Move timer code to separate file
  ARM: ux500: add support for clocksource DBX500 PRCMU
  clocksource: add DBX500 PRCMU Timer support
  ARM: plat-nomadik: MTU sched_clock as an option
  ARM: OMAP: dmtimer: add error handling to export APIs
  ARM: OMAP: dmtimer: low-power mode support
  ARM: OMAP: dmtimer: skip reserved timers
  ARM: OMAP: dmtimer: pm_runtime support
  ARM: OMAP: dmtimer: switch-over to platform device driver
  ARM: OMAP: dmtimer: platform driver
  ARM: OMAP2+: dmtimer: convert to platform devices
  ARM: OMAP1: dmtimer: conversion to platform devices
  ARM: OMAP2+: dmtimer: add device names to flck nodes
  ARM: OMAP: Add support for dmtimer v2 ip

* 'next/driver' of git://git.linaro.org/people/arnd/arm-soc:
  hw_random: add driver for atmel true hardware random number generator
  ARM: at91: at91sam9g45: add trng clock and platform device
  MX53 Enable the AHCI SATA on MX53 SMD board
  MX53 Enable the AHCI SATA on MX53 LOCO board
  MX53 Enable the AHCI SATA on MX53 ARD board
  AHCI Add the AHCI SATA feature on the MX53 platforms
  Fix pata imx resource
  ARM: imx: Define functions for registering PATA
  ARM: imx: Add PATA clock support
  ARM: imx: Add PATA resources for other i.MX processors
  imx: efika: Enable pata.
  imx51: add pata clock
  imx51: add pata device

Fix up trivial conflict (new selects next to each other from separate
branches for EFIKA_COMMON) in arch/arm/mach-mx5/Kconfig

* 'next/cleanup' of git://git.linaro.org/people/arnd/arm-soc: (125 commits)
  ARM: mach-mxs: fix machines' initializers order
  mmc: mxcmmc: explicitly includes mach/hardware.h
  arm/imx: explicitly includes mach/hardware.h in pm-imx27.c
  arm/imx: remove mx27_setup_weimcs() from mx27.h
  arm/imx: explicitly includes mach/hardware.h in mach-kzm_arm11_01.c
  arm/imx: remove mx31_setup_weimcs() from mx31.h
  ARM: tegra: devices.c should include devices.h
  ARM: tegra: cpu-tegra: unexport two functions
  ARM: tegra: cpu-tegra: sparse type fix
  ARM: tegra: dma: staticify some tables and functions
  ARM: tegra: tegra2_clocks: don't export some tables
  ARM: tegra: tegra_powergate_is_powered should be static
  ARM: tegra: tegra_rtc_read_ms should be static
  ARM: tegra: tegra_init_cache should be static
  ARM: tegra: pcie: 0 -> NULL changes
  ARM: tegra: pcie: include board.h
  ARM: tegra: pcie: don't cast __iomem pointers
  ARM: tegra: tegra2_clocks: 0 -> NULL changes
  ARM: tegra: tegra2_clocks: don't cast __iomem pointers
  ARM: tegra: timer: don't cast __iomem pointers
  ...

Fix up trivial conflicts in
  arch/arm/mach-omap2/Makefile,
  arch/arm/mach-u300/{Makefile.boot,core.c}
  arch/arm/plat-{mxc,omap}/devices.c

* 'next/fixes' of git://git.linaro.org/people/arnd/arm-soc: (28 commits)
  ARM: pxa/cm-x300: properly set bt_reset pin
  ARM: mmp: rename SHEEVAD to GPLUGD
  ARM: imx: Fix typo 'MACH_MX31_3DS_MXC_NAND_USE_BBT'
  ARM: i.MX28: shift frac value in _CLK_SET_RATE
  plat-mxc: iomux-v3.h: implicitly enable pull-up/down when that's desired
  ARM: mx5: fix clock usage for suspend
  ARM: pxa: use correct __iomem annotations
  ARM: pxa: sharpsl pm needs SPI
  ARM: pxa: centro and treo680 need palm27x
  ARM: pxa: make pxafb_smart_*() empty when not enabled
  ARM: pxa: select POWER_SUPPLY on raumfeld
  ARM: pxa: pxa95x is incompatible with earlier pxa
  ARM: pxa: CPU_FREQ_TABLE is needed for CPU_FREQ
  ARM: pxa: pxa95x/saarb depends on pxa3xx code
  ARM: pxa: allow selecting just one of TREO680/CENTRO
  ARM: pxa: export symbols from pxa3xx-ulpi
  ARM: pxa: make zylonite_pxa*_init declaration match code
  ARM: pxa/z2: fix building error of pxa27x_cpu_suspend() no longer available
  ARM: at91: add defconfig for at91sam9g45 family
  ARM: at91: remove dependency for Atmel PWM driver selector in Kconfig
  ...

Conflicts:
	Documentation/devicetree/bindings/arm/fsl.txt
	arch/arm/Kconfig
	arch/arm/Kconfig.debug
	arch/arm/plat-mxc/include/mach/common.h