We avoid evicting and scanning anonymous pages for the most part, but
under some workloads we can end up with most of memory filled with
anonymous pages.  At that point, we suddenly need to clear the referenced
bits on all of memory, which can take ages on very large memory systems.

We can reduce the maximum number of pages that need to be scanned by not
taking the referenced state into account when deactivating an anonymous
page.  After all, every anonymous page starts out referenced, so why
check?

If an anonymous page gets referenced again before it reaches the end of
the inactive list, we move it back to the active list.

To keep the maximum amount of necessary work reasonable, we scale the
active to inactive ratio with the size of memory, using the formula
active:inactive ratio = sqrt(memory in GB * 10).

Kswapd CPU use now seems to scale by the amount of pageout bandwidth,
instead of by the amount of memory present in the system.

[kamezawa.hiroyu@jp.fujitsu.com: fix OOM with memcg]
[kamezawa.hiroyu@jp.fujitsu.com: memcg: lru scan fix]
Signed-off-by: NRik van Riel <riel@redhat.com>
Signed-off-by: NKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Signed-off-by: NKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Split the LRU lists in two, one set for pages that are backed by real file
systems ("file") and one for pages that are backed by memory and swap
("anon").  The latter includes tmpfs.

The advantage of doing this is that the VM will not have to scan over lots
of anonymous pages (which we generally do not want to swap out), just to
find the page cache pages that it should evict.

This patch has the infrastructure and a basic policy to balance how much
we scan the anon lists and how much we scan the file lists.  The big
policy changes are in separate patches.

[lee.schermerhorn@hp.com: collect lru meminfo statistics from correct offset]
[kosaki.motohiro@jp.fujitsu.com: prevent incorrect oom under split_lru]
[kosaki.motohiro@jp.fujitsu.com: fix pagevec_move_tail() doesn't treat unevictable page]
[hugh@veritas.com: memcg swapbacked pages active]
[hugh@veritas.com: splitlru: BDI_CAP_SWAP_BACKED]
[akpm@linux-foundation.org: fix /proc/vmstat units]
[nishimura@mxp.nes.nec.co.jp: memcg: fix handling of shmem migration]
[kosaki.motohiro@jp.fujitsu.com: adjust Quicklists field of /proc/meminfo]
[kosaki.motohiro@jp.fujitsu.com: fix style issue of get_scan_ratio()]
Signed-off-by: NRik van Riel <riel@redhat.com>
Signed-off-by: NLee Schermerhorn <Lee.Schermerhorn@hp.com>
Signed-off-by: NKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Signed-off-by: NHugh Dickins <hugh@veritas.com>
Signed-off-by: NDaisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Define page_file_cache() function to answer the question:
	is page backed by a file?

Originally part of Rik van Riel's split-lru patch.  Extracted to make
available for other, independent reclaim patches.

Moved inline function to linux/mm_inline.h where it will be needed by
subsequent "split LRU" and "noreclaim" patches.

Unfortunately this needs to use a page flag, since the PG_swapbacked state
needs to be preserved all the way to the point where the page is last
removed from the LRU.  Trying to derive the status from other info in the
page resulted in wrong VM statistics in earlier split VM patchsets.

The total number of page flags in use on a 32 bit machine after this patch
is 19.

[akpm@linux-foundation.org: fix up out-of-order merge fallout]
[hugh@veritas.com: splitlru: shmem_getpage SetPageSwapBacked sooner[
Signed-off-by: NRik van Riel <riel@redhat.com>
Signed-off-by: NLee Schermerhorn <lee.schermerhorn@hp.com>
Signed-off-by: NMinChan Kim <minchan.kim@gmail.com>
Signed-off-by: NHugh Dickins <hugh@veritas.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

If vm_swap_full() (swap space more than 50% full), the system will free
swap space at swapin time.  With this patch, the system will also free the
swap space in the pageout code, when we decide that the page is not a
candidate for swapout (and just wasting swap space).
Signed-off-by: NRik van Riel <riel@redhat.com>
Signed-off-by: NLee Schermerhorn <Lee.Schermerhorn@hp.com>
Signed-off-by: NMinChan Kim <minchan.kim@gmail.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Turn the pagevecs into an array just like the LRUs.  This significantly
cleans up the source code and reduces the size of the kernel by about 13kB
after all the LRU lists have been created further down in the split VM
patch series.
Signed-off-by: NRik van Riel <riel@redhat.com>
Signed-off-by: NKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Currently we are defining explicit variables for the inactive and active
list.  An indexed array can be more generic and avoid repeating similar
code in several places in the reclaim code.

We are saving a few bytes in terms of code size:

Before:

   text    data     bss     dec     hex filename
4097753  573120 4092484 8763357  85b7dd vmlinux

After:

   text    data     bss     dec     hex filename
4097729  573120 4092484 8763333  85b7c5 vmlinux

Having an easy way to add new lru lists may ease future work on the
reclaim code.
Signed-off-by: NRik van Riel <riel@redhat.com>
Signed-off-by: NLee Schermerhorn <lee.schermerhorn@hp.com>
Signed-off-by: NChristoph Lameter <cl@linux-foundation.org>
Signed-off-by: NKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

On large memory systems, the VM can spend way too much time scanning
through pages that it cannot (or should not) evict from memory.  Not only
does it use up CPU time, but it also provokes lock contention and can
leave large systems under memory presure in a catatonic state.

This patch series improves VM scalability by:

1) putting filesystem backed, swap backed and unevictable pages
   onto their own LRUs, so the system only scans the pages that it
   can/should evict from memory

2) switching to two handed clock replacement for the anonymous LRUs,
   so the number of pages that need to be scanned when the system
   starts swapping is bound to a reasonable number

3) keeping unevictable pages off the LRU completely, so the
   VM does not waste CPU time scanning them. ramfs, ramdisk,
   SHM_LOCKED shared memory segments and mlock()ed VMA pages
   are keept on the unevictable list.

This patch:

isolate_lru_page logically belongs to be in vmscan.c than migrate.c.

It is tough, because we don't need that function without memory migration
so there is a valid argument to have it in migrate.c.  However a
subsequent patch needs to make use of it in the core mm, so we can happily
move it to vmscan.c.

Also, make the function a little more generic by not requiring that it
adds an isolated page to a given list.  Callers can do that.

	Note that we now have '__isolate_lru_page()', that does
	something quite different, visible outside of vmscan.c
	for use with memory controller.  Methinks we need to
	rationalize these names/purposes.	--lts

[akpm@linux-foundation.org: fix mm/memory_hotplug.c build]
Signed-off-by: NNick Piggin <npiggin@suse.de>
Signed-off-by: NRik van Riel <riel@redhat.com>
Signed-off-by: NLee Schermerhorn <Lee.Schermerhorn@hp.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

There is nothing architecture specific about remove_memory().
remove_memory() function is common for all architectures which support
hotplug memory remove.  Instead of duplicating it in every architecture,
collapse them into arch neutral function.

[akpm@linux-foundation.org: fix the export]
Signed-off-by: NBadari Pulavarty <pbadari@us.ibm.com>
Cc: Yasunori Goto <y-goto@jp.fujitsu.com>
Cc: Gary Hade <garyhade@us.ibm.com>
Cc: Mel Gorman <mel@csn.ul.ie>
Cc: Yasunori Goto <y-goto@jp.fujitsu.com>
Cc: "Luck, Tony" <tony.luck@intel.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Fix a warning caused by commit 0c8946d9
(serial: Make uart_port's ioport "unsigned long".)
Signed-off-by: NAtsushi Nemoto <anemo@mba.ocn.ne.jp>
Cc: Alan Cox <alan@lxorguk.ukuu.org.uk>
Cc: Josip Rodin <joy@entuzijast.net>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Today's linux-next build (powerpc_allyesconfig) failed like this:

drivers/char/tpm/tpm.c:1162: error: __ksymtab_tpm_dev_release causes a section type conflict

Caused by commit 253115b7 ("The
tpm_dev_release function is only called for platform devices, not pnp")
which exported a static function.
Signed-off-by: NStephen Rothwell <sfr@canb.auug.org.au>
Cc: Rajiv Andrade <srajiv@linux.vnet.ibm.com>
Cc: James Morris <jmorris@namei.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

A couple of commits have a broken real name - fix them up.
Signed-off-by: NMark Brown <broonie@sirena.org.uk>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

The anon_vma code is very subtle, and we end up doing optimistic lookups
of anon_vmas under RCU in page_lock_anon_vma() with no locking.  Other
CPU's can also see the newly allocated entry immediately after we've
exposed it by setting "vma->anon_vma" to the new value.

We protect against the anon_vma being destroyed by having the SLAB
marked as SLAB_DESTROY_BY_RCU, so the RCU lookup can depend on the
allocation not being destroyed - but it might still be free'd and
re-allocated here to a new vma.

As a result, we should not do the anon_vma list ops on a newly allocated
vma without proper locking.
Acked-by: NNick Piggin <npiggin@suse.de>
Acked-by: NHugh Dickins <hugh@veritas.com>
Acked-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

* git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb-2.6: (94 commits)
  USB: remove err() macro from more usb drivers
  USB: remove err() macro from usb misc drivers
  USB: remove err() macro from usb core code
  USB: remove err() macro from usb class drivers
  USB: remove use of err() in drivers/usb/serial
  USB: remove info() macro from usb mtd drivers
  USB: remove info() macro from usb input drivers
  USB: remove info() macro from usb network drivers
  USB: remove info() macro from remaining usb drivers
  USB: remove info() macro from usb/misc drivers
  USB: remove info() macro from usb/serial drivers
  USB: remove warn macro from HID core
  USB: remove warn() macro from usb drivers
  USB: remove warn() macro from usb net drivers
  USB: remove warn() macro from usb media drivers
  USB: remove warn() macro from usb input drivers
  usb/fsl_qe_udc: clear data toggle on clear halt request
  usb/fsl_qe_udc: fix response to get status request
  fsl_usb2_udc: Fix oops on probe failure.
  fsl_usb2_udc: Add a wmb before priming endpoint.
  ...

* 'drm-next' of git://git.kernel.org/pub/scm/linux/kernel/git/airlied/drm-2.6: (44 commits)
  drm/i915: fix ioremap of a user address for non-root (CVE-2008-3831)
  drm: make CONFIG_DRM depend on CONFIG_SHMEM.
  radeon: fix PCI bus mastering support enables.
  radeon: add RS400 family support.
  drm/radeon: add support for RS740 IGP chipsets.
  i915: GM45 has GM965-style MCH setup.
  i915: Don't run retire work handler while suspended
  i915: Map status page cached for chips with GTT-based HWS location.
  i915: Fix up ring initialization to cover G45 oddities
  i915: Use non-reserved status page index for breadcrumb
  drm: Increment dev_priv->irq_received so i915_gem_interrupts count works.
  drm: kill drm_device->irq
  drm: wbinvd is cache coherent.
  i915: add missing return in error path.
  i915: fixup permissions on gem ioctls.
  drm: Clean up many sparse warnings in i915.
  drm: Use ioremap_wc in i915_driver instead of ioremap, since we always want WC.
  drm: G33-class hardware has a newer 965-style MCH (no DCC register).
  drm: Avoid oops in GEM execbuffers with bad arguments.
  DRM: Return -EBADF on bad object in flink, and return curent name if it exists.
  ...

* 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-2.6: (95 commits)
  V4L/DVB (9296): Patch to remove warning message during cx88-dvb compilation
  V4L/DVB (9294): gspca: Add a stop sequence in t613.
  V4L/DVB (9293): gspca: Separate and fix the sensor dependant sequences in t613.
  V4L/DVB (9292): gspca: Call the control setting functions at init time in t613.
  V4L/DVB (9291): gspca: Do not set the white balance temperature by default in t613.
  V4L/DVB (9290): gspca: Adjust the sensor init sequences in t613.
  V4L/DVB (9289): gspca: Other sensor identified as om6802 in t613.
  V4L/DVB (9288): gspca: Write to the USB device and not USB interface in t613.
  V4L/DVB (9287): gspca: Change the name of the multi bytes write function in t613.
  V4L/DVB (9286): gspca: Compilation problem of gspca.c and the kernel version.
  V4L/DVB (9283): Correct typo and enable setting the gain on the mt9m111 sensor
  V4L/DVB (9282): Properly iterate the urbs when destroying them.
  V4L/DVB (9281): gspca: Add hflip and vflip to the po1030 sensor
  V4L/DVB (9280): gspca: Use the gspca debug macros
  V4L/DVB (9279): gspca: Correct some copyright headers
  V4L/DVB (9278): gspca: Remove the m5602_debug variable
  V4L/DVB (9277): gspca: propagate an error in m5602_start_transfer()
  V4L/DVB (9276): videobuf-dvb: two functions are now static
  V4L/DVB (9275): dvb: input data pointer of cx24116_writeregN() should be const
  V4L/DVB (9274): Remove spurious messages and turn into debug.
  ...

* 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4:
  ext4: Remove automatic enabling of the HUGE_FILE feature flag
  ext4: Replace hackish ext4_mb_poll_new_transaction with commit callback
  ext4: Update Documentation/filesystems/ext4.txt
  ext4: Remove unused mount options: nomballoc, mballoc, nocheck
  ext4: Remove compile warnings when building w/o CONFIG_PROC_FS
  ext4: Add missing newlines to printk messages
  ext4: Fix file fragmentation during large file write.
  vfs: Add no_nrwrite_index_update writeback control flag
  vfs: Remove the range_cont writeback mode.
  ext4: Use tag dirty lookup during mpage_da_submit_io
  ext4: let the block device know when unused blocks can be discarded
  ext4: Don't reuse released data blocks until transaction commits
  ext4: Use an rbtree for tracking blocks freed during transaction.
  ext4: Do mballoc init before doing filesystem recovery
  ext4: Free ext4_prealloc_space using kmem_cache_free
  ext4: Fix Kconfig typo for ext4dev
  ext4: Remove an old reference to ext4dev in Makefile comment

USB should not be having it's own printk macros, so remove err() and
use the system-wide standard of dev_err() wherever possible.  In the
few places that will not work out, use a basic printk().
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

USB should not be having it's own printk macros, so remove err() and
use the system-wide standard of dev_err() wherever possible.  In the
few places that will not work out, use a basic printk().
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

USB should not be having it's own printk macros, so remove err() and
use the system-wide standard of dev_err() wherever possible.  In the
few places that will not work out, use a basic printk().
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

USB should not be having it's own printk macros, so remove err() and
use the system-wide standard of dev_err() wherever possible.  In the
few places that will not work out, use a basic printk().
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

err() is going away, so switch to dev_err() or printk() if it's really
needed.
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

USB should not be having it's own printk macros, so remove info() and
use the system-wide standard of dev_info() wherever possible.
Acked-by: NDavid Woodhouse <dwmw2@infradead.org>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

USB should not be having it's own printk macros, so remove info() and
use the system-wide standard of dev_info() wherever possible.
Acked-by: NDmitry Torokhov <dmitry.torokhov@gmail.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

USB should not be having it's own printk macros, so remove info() and
use the system-wide standard of dev_info() wherever possible.

Cc: Jeff Garzik <jgarzik@pobox.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

USB should not be having it's own printk macros, so remove info() and
use the system-wide standard of dev_info() wherever possible.  In the
few places that will not work out, use a basic printk().

Clean up the remaining usages of this in the drivers/usb/ directory.
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

USB should not be having it's own printk macros, so remove info() and
use the system-wide standard of dev_info() wherever possible.
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

USB should not be having it's own printk macros, so remove info() and
use the system-wide standard of dev_info() wherever possible.
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

There were two stragglers that got missed in the last merge of the HID tree that forgot to change the warn() calls to dev_warn().  This patch fixes them up.
Acked-by: NJiri Kosina <jkosina@suse.cz>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

USB should not be having it's own printk macros, so remove warn() and
use the system-wide standard of dev_warn() wherever possible.  In the
few places that will not work out, use a basic printk().
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

USB should not be having it's own printk macros, so remove warn() and
use the system-wide standard of dev_warn() wherever possible.  In the
few places that will not work out, use a basic printk().

Cc: Jeff Garzik <jgarzik@pobox.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

USB should not be having it's own printk macros, so remove warn() and
use the system-wide standard of dev_warn() wherever possible.  In the
few places that will not work out, use a basic printk().

Cc: Mauro Carvalho Chehab <mchehab@infradead.org>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

USB should not be having it's own printk macros, so remove warn() and
use the system-wide standard of dev_warn() wherever possible.  In the
few places that will not work out, use a basic printk().

Cc: Dmitry Torokhov <dmitry.torokhov@gmail.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

Fix to comply with USB spec.
Signed-off-by: NLi Yang <leoli@freescale.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

The original code didn't respond correctly to get status request on
device and endpoint.  Although normal operations can work without the
fix.  It is not compliant with USB spec chapter9 and fails USBCV ch9
tests.  The patch fix this and a few style/typo problems.
Signed-off-by: NLi Yang <leoli@freescale.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

In some circumstances when fsl_udc_probe fails udc_controller is freed but
the pointer remains non-NULL. fsl_udc_remove will then try and teardown
the partly initialized and freed controller structure resulting in an oops.
This patch ensures udc_controller is either NULL or fully initialized after
fsl_udc_probe.
Signed-off-by: NWill Newton <will.newton@gmail.com>
Acked-by: NLi Yang <leoli@freescale.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

Add a wmb to fsl_queue_td before priming the endpoint. This ensures that the
modifications to the QH are seen by the hardware.

Added comment as suggested by Felipe Balbi.
Signed-off-by: NWill Newton <will.newton@gmail.com>
Acked-by: NLi Yang <leoli@freescale.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

fsl_queue_td always returns 0. Make it void and remove checks for non-zero
return in callers.
Signed-off-by: NWill Newton <will.newton@gmail.com>
Acked-by: NLi Yang <leoli@freescale.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

Uninline udc_reset_ep_queue and remove it's unused return value.
Signed-off-by: NWill Newton <will.newton@gmail.com>
Acked-by: NLi Yang <leoli@freescale.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

Rename the arguments of the fsl_writel macro to match their use.
Remove a couple of unnecessary prototypes.
Signed-off-by: NWill Newton <will.newton@gmail.com>
Acked-by: NLi Yang <leoli@freescale.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

Move spinlock initialization earlier so we can turn shared irq handler
debugging on safely.
Signed-off-by: NWill Newton <will.newton@gmail.com>
Acked-by: NLi Yang <leoli@freescale.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>