Skip to content

K
Kernel

openeuler / Kernel
1 年多 前同步成功

通知 8
Star 0
Fork 0
K
Kernel
切换分支/标签
  1. 04 9月, 2021 28 次提交
    • S
      mm: slub: move flush_cpu_slab() invocations __free_slab() invocations out of IRQ context · 5a836bf6
      由 Sebastian Andrzej Siewior 提交于 
      flush_all() flushes a specific SLAB cache on each CPU (where the cache
is present). The deactivate_slab()/__free_slab() invocation happens
within IPI handler and is problematic for PREEMPT_RT.

The flush operation is not a frequent operation or a hot path. The
per-CPU flush operation can be moved to within a workqueue.

Because a workqueue handler, unlike IPI handler, does not disable irqs,
flush_slab() now has to disable them for working with the kmem_cache_cpu
fields. deactivate_slab() is safe to call with irqs enabled.

[vbabka@suse.cz: adapt to new SLUB changes]
Signed-off-by: NSebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      5a836bf6
    • V
      mm, slab: split out the cpu offline variant of flush_slab() · 08beb547
      由 Vlastimil Babka 提交于 
      flush_slab() is called either as part IPI handler on given live cpu, or as a
cleanup on behalf of another cpu that went offline. The first case needs to
protect updating the kmem_cache_cpu fields with disabled irqs. Currently the
whole call happens with irqs disabled by the IPI handler, but the following
patch will change from IPI to workqueue, and flush_slab() will have to disable
irqs (to be replaced with a local lock later) in the critical part.

To prepare for this change, replace the call to flush_slab() for the dead cpu
handling with an opencoded variant that will not disable irqs nor take a local
lock.
Suggested-by: NMike Galbraith <efault@gmx.de>
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      08beb547
    • V
      mm, slub: don't disable irqs in slub_cpu_dead() · 0e7ac738
      由 Vlastimil Babka 提交于 
      slub_cpu_dead() cleans up for an offlined cpu from another cpu and calls only
functions that are now irq safe, so we don't need to disable irqs anymore.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      0e7ac738
    • V
      mm, slub: only disable irq with spin_lock in __unfreeze_partials() · 7cf9f3ba
      由 Vlastimil Babka 提交于 
      __unfreeze_partials() no longer needs to have irqs disabled, except for making
the spin_lock operations irq-safe, so convert the spin_locks operations and
remove the separate irq handling.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      7cf9f3ba
    • V
      mm, slub: separate detaching of partial list in unfreeze_partials() from unfreezing · fc1455f4
      由 Vlastimil Babka 提交于 
      Unfreezing partial list can be split to two phases - detaching the list from
struct kmem_cache_cpu, and processing the list. The whole operation does not
need to be protected by disabled irqs. Restructure the code to separate the
detaching (with disabled irqs) and unfreezing (with irq disabling to be reduced
in the next patch).

Also, unfreeze_partials() can be called from another cpu on behalf of a cpu
that is being offlined, where disabling irqs on the local cpu has no sense, so
restructure the code as follows:

- __unfreeze_partials() is the bulk of unfreeze_partials() that processes the
  detached percpu partial list
- unfreeze_partials() detaches list from current cpu with irqs disabled and
  calls __unfreeze_partials()
- unfreeze_partials_cpu() is to be called for the offlined cpu so it needs no
  irq disabling, and is called from __flush_cpu_slab()
- flush_cpu_slab() is for the local cpu thus it needs to call
  unfreeze_partials(). So it can't simply call
  __flush_cpu_slab(smp_processor_id()) anymore and we have to open-code the
  proper calls.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      fc1455f4
    • V
      mm, slub: detach whole partial list at once in unfreeze_partials() · c2f973ba
      由 Vlastimil Babka 提交于 
      Instead of iterating through the live percpu partial list, detach it from the
kmem_cache_cpu at once. This is simpler and will allow further optimization.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      c2f973ba
    • V
      mm, slub: discard slabs in unfreeze_partials() without irqs disabled · 8de06a6f
      由 Vlastimil Babka 提交于 
      No need for disabled irqs when discarding slabs, so restore them before
discarding.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      8de06a6f
    • V
      mm, slub: move irq control into unfreeze_partials() · f3ab8b6b
      由 Vlastimil Babka 提交于 
      unfreeze_partials() can be optimized so that it doesn't need irqs disabled for
the whole time. As the first step, move irq control into the function and
remove it from the put_cpu_partial() caller.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      f3ab8b6b
    • V
      mm, slub: call deactivate_slab() without disabling irqs · cfdf836e
      由 Vlastimil Babka 提交于 
      The function is now safe to be called with irqs enabled, so move the calls
outside of irq disabled sections.

When called from ___slab_alloc() -> flush_slab() we have irqs disabled, so to
reenable them before deactivate_slab() we need to open-code flush_slab() in
___slab_alloc() and reenable irqs after modifying the kmem_cache_cpu fields.
But that means a IRQ handler meanwhile might have assigned a new page to
kmem_cache_cpu.page so we have to retry the whole check.

The remaining callers of flush_slab() are the IPI handler which has disabled
irqs anyway, and slub_cpu_dead() which will be dealt with in the following
patch.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      cfdf836e
    • V
      mm, slub: make locking in deactivate_slab() irq-safe · 3406e91b
      由 Vlastimil Babka 提交于 
      dectivate_slab() now no longer touches the kmem_cache_cpu structure, so it will
be possible to call it with irqs enabled. Just convert the spin_lock calls to
their irq saving/restoring variants to make it irq-safe.

Note we now have to use cmpxchg_double_slab() for irq-safe slab_lock(), because
in some situations we don't take the list_lock, which would disable irqs.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      3406e91b
    • V
      mm, slub: move reset of c->page and freelist out of deactivate_slab() · a019d201
      由 Vlastimil Babka 提交于 
      deactivate_slab() removes the cpu slab by merging the cpu freelist with slab's
freelist and putting the slab on the proper node's list. It also sets the
respective kmem_cache_cpu pointers to NULL.

By extracting the kmem_cache_cpu operations from the function, we can make it
not dependent on disabled irqs.

Also if we return a single free pointer from ___slab_alloc, we no longer have
to assign kmem_cache_cpu.page before deactivation or care if somebody preempted
us and assigned a different page to our kmem_cache_cpu in the process.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      a019d201
    • V
      mm, slub: stop disabling irqs around get_partial() · 4b1f449d
      由 Vlastimil Babka 提交于 
      The function get_partial() does not need to have irqs disabled as a whole. It's
sufficient to convert spin_lock operations to their irq saving/restoring
versions.

As a result, it's now possible to reach the page allocator from the slab
allocator without disabling and re-enabling interrupts on the way.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      4b1f449d
    • V
      mm, slub: check new pages with restored irqs · 9f101ee8
      由 Vlastimil Babka 提交于 
      Building on top of the previous patch, re-enable irqs before checking new
pages. alloc_debug_processing() is now called with enabled irqs so we need to
remove VM_BUG_ON(!irqs_disabled()); in check_slab() - there doesn't seem to be
a need for it anyway.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      9f101ee8
    • V
      mm, slub: validate slab from partial list or page allocator before making it cpu slab · 3f2b77e3
      由 Vlastimil Babka 提交于 
      When we obtain a new slab page from node partial list or page allocator, we
assign it to kmem_cache_cpu, perform some checks, and if they fail, we undo
the assignment.

In order to allow doing the checks without irq disabled, restructure the code
so that the checks are done first, and kmem_cache_cpu.page assignment only
after they pass.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      3f2b77e3
    • V
      mm, slub: restore irqs around calling new_slab() · 6c1dbb67
      由 Vlastimil Babka 提交于 
      allocate_slab() currently re-enables irqs before calling to the page allocator.
It depends on gfpflags_allow_blocking() to determine if it's safe to do so.
Now we can instead simply restore irq before calling it through new_slab().
The other caller early_kmem_cache_node_alloc() is unaffected by this.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      6c1dbb67
    • V
      mm, slub: move disabling irqs closer to get_partial() in ___slab_alloc() · fa417ab7
      由 Vlastimil Babka 提交于 
      Continue reducing the irq disabled scope. Check for per-cpu partial slabs with
first with irqs enabled and then recheck with irqs disabled before grabbing
the slab page. Mostly preparatory for the following patches.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      fa417ab7
    • V
      mm, slub: do initial checks in ___slab_alloc() with irqs enabled · 0b303fb4
      由 Vlastimil Babka 提交于 
      As another step of shortening irq disabled sections in ___slab_alloc(), delay
disabling irqs until we pass the initial checks if there is a cached percpu
slab and it's suitable for our allocation.

Now we have to recheck c->page after actually disabling irqs as an allocation
in irq handler might have replaced it.

Because we call pfmemalloc_match() as one of the checks, we might hit
VM_BUG_ON_PAGE(!PageSlab(page)) in PageSlabPfmemalloc in case we get
interrupted and the page is freed. Thus introduce a pfmemalloc_match_unsafe()
variant that lacks the PageSlab check.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
Acked-by: NMel Gorman <mgorman@techsingularity.net>
      0b303fb4
    • V
      mm, slub: move disabling/enabling irqs to ___slab_alloc() · e500059b
      由 Vlastimil Babka 提交于 
      Currently __slab_alloc() disables irqs around the whole ___slab_alloc().  This
includes cases where this is not needed, such as when the allocation ends up in
the page allocator and has to awkwardly enable irqs back based on gfp flags.
Also the whole kmem_cache_alloc_bulk() is executed with irqs disabled even when
it hits the __slab_alloc() slow path, and long periods with disabled interrupts
are undesirable.

As a first step towards reducing irq disabled periods, move irq handling into
___slab_alloc(). Callers will instead prevent the s->cpu_slab percpu pointer
from becoming invalid via get_cpu_ptr(), thus preempt_disable(). This does not
protect against modification by an irq handler, which is still done by disabled
irq for most of ___slab_alloc(). As a small immediate benefit,
slab_out_of_memory() from ___slab_alloc() is now called with irqs enabled.

kmem_cache_alloc_bulk() disables irqs for its fastpath and then re-enables them
before calling ___slab_alloc(), which then disables them at its discretion. The
whole kmem_cache_alloc_bulk() operation also disables preemption.

When  ___slab_alloc() calls new_slab() to allocate a new page, re-enable
preemption, because new_slab() will re-enable interrupts in contexts that allow
blocking (this will be improved by later patches).

The patch itself will thus increase overhead a bit due to disabled preemption
(on configs where it matters) and increased disabling/enabling irqs in
kmem_cache_alloc_bulk(), but that will be gradually improved in the following
patches.

Note in __slab_alloc() we need to change the #ifdef CONFIG_PREEMPT guard to
CONFIG_PREEMPT_COUNT to make sure preempt disable/enable is properly paired in
all configurations. On configs without involuntary preemption and debugging
the re-read of kmem_cache_cpu pointer is still compiled out as it was before.

[ Mike Galbraith <efault@gmx.de>: Fix kmem_cache_alloc_bulk() error path ]
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      e500059b
    • V
      mm, slub: simplify kmem_cache_cpu and tid setup · 9b4bc85a
      由 Vlastimil Babka 提交于 
      In slab_alloc_node() and do_slab_free() fastpaths we need to guarantee that
our kmem_cache_cpu pointer is from the same cpu as the tid value. Currently
that's done by reading the tid first using this_cpu_read(), then the
kmem_cache_cpu pointer and verifying we read the same tid using the pointer and
plain READ_ONCE().

This can be simplified to just fetching kmem_cache_cpu pointer and then reading
tid using the pointer. That guarantees they are from the same cpu. We don't
need to read the tid using this_cpu_read() because the value will be validated
by this_cpu_cmpxchg_double(), making sure we are on the correct cpu and the
freelist didn't change by anyone preempting us since reading the tid.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
Acked-by: NMel Gorman <mgorman@techsingularity.net>
      9b4bc85a
    • V
      mm, slub: restructure new page checks in ___slab_alloc() · 1572df7c
      由 Vlastimil Babka 提交于 
      When we allocate slab object from a newly acquired page (from node's partial
list or page allocator), we usually also retain the page as a new percpu slab.
There are two exceptions - when pfmemalloc status of the page doesn't match our
gfp flags, or when the cache has debugging enabled.

The current code for these decisions is not easy to follow, so restructure it
and add comments. The new structure will also help with the following changes.
No functional change.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
Acked-by: NMel Gorman <mgorman@techsingularity.net>
      1572df7c
    • V
      mm, slub: return slab page from get_partial() and set c->page afterwards · 75c8ff28
      由 Vlastimil Babka 提交于 
      The function get_partial() finds a suitable page on a partial list, acquires
and returns its freelist and assigns the page pointer to kmem_cache_cpu.
In later patch we will need more control over the kmem_cache_cpu.page
assignment, so instead of passing a kmem_cache_cpu pointer, pass a pointer to a
pointer to a page that get_partial() can fill and the caller can assign the
kmem_cache_cpu.page pointer. No functional change as all of this still happens
with disabled IRQs.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      75c8ff28
    • V
      mm, slub: dissolve new_slab_objects() into ___slab_alloc() · 53a0de06
      由 Vlastimil Babka 提交于 
      The later patches will need more fine grained control over individual actions
in ___slab_alloc(), the only caller of new_slab_objects(), so dissolve it
there. This is a preparatory step with no functional change.

The only minor change is moving WARN_ON_ONCE() for using a constructor together
with __GFP_ZERO to new_slab(), which makes it somewhat less frequent, but still
able to catch a development change introducing a systematic misuse.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
Acked-by: NChristoph Lameter <cl@linux.com>
Acked-by: NMel Gorman <mgorman@techsingularity.net>
      53a0de06
    • V
      mm, slub: extract get_partial() from new_slab_objects() · 2a904905
      由 Vlastimil Babka 提交于 
      The later patches will need more fine grained control over individual actions
in ___slab_alloc(), the only caller of new_slab_objects(), so this is a first
preparatory step with no functional change.

This adds a goto label that appears unnecessary at this point, but will be
useful for later changes.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
Acked-by: NChristoph Lameter <cl@linux.com>
      2a904905
    • V
      mm, slub: remove redundant unfreeze_partials() from put_cpu_partial() · 976b805c
      由 Vlastimil Babka 提交于 
      Commit d6e0b7fa ("slub: make dead caches discard free slabs immediately")
introduced cpu partial flushing for kmemcg caches, based on setting the target
cpu_partial to 0 and adding a flushing check in put_cpu_partial().
This code that sets cpu_partial to 0 was later moved by c9fc5864 ("slab:
introduce __kmemcg_cache_deactivate()") and ultimately removed by 9855609b
("mm: memcg/slab: use a single set of kmem_caches for all accounted
allocations"). However the check and flush in put_cpu_partial() was never
removed, although it's effectively a dead code. So this patch removes it.

Note that d6e0b7fa also added preempt_disable()/enable() to
unfreeze_partials() which could be thus also considered unnecessary. But
further patches will rely on it, so keep it.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
      976b805c
    • V
      mm, slub: don't disable irq for debug_check_no_locks_freed() · 84048039
      由 Vlastimil Babka 提交于 
      In slab_free_hook() we disable irqs around the debug_check_no_locks_freed()
call, which is unnecessary, as irqs are already being disabled inside the call.
This seems to be leftover from the past where there were more calls inside the
irq disabled sections. Remove the irq disable/enable operations.

Mel noted:
> Looks like it was needed for kmemcheck which went away back in 4.15
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
Acked-by: NMel Gorman <mgorman@techsingularity.net>
      84048039
    • V
      mm, slub: allocate private object map for validate_slab_cache() · 0a19e7dd
      由 Vlastimil Babka 提交于 
      validate_slab_cache() is called either to handle a sysfs write, or from a
self-test context. In both situations it's straightforward to preallocate a
private object bitmap instead of grabbing the shared static one meant for
critical sections, so let's do that.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
Acked-by: NChristoph Lameter <cl@linux.com>
Acked-by: NMel Gorman <mgorman@techsingularity.net>
      0a19e7dd
    • V
      mm, slub: allocate private object map for debugfs listings · b3fd64e1
      由 Vlastimil Babka 提交于 
      Slub has a static spinlock protected bitmap for marking which objects are on
freelist when it wants to list them, for situations where dynamically
allocating such map can lead to recursion or locking issues, and on-stack
bitmap would be too large.

The handlers of debugfs files alloc_traces and free_traces also currently use this
shared bitmap, but their syscall context makes it straightforward to allocate a
private map before entering locked sections, so switch these processing paths
to use a private bitmap.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
Acked-by: NChristoph Lameter <cl@linux.com>
Acked-by: NMel Gorman <mgorman@techsingularity.net>
      b3fd64e1
    • V
      mm, slub: don't call flush_all() from slab_debug_trace_open() · eafb1d64
      由 Vlastimil Babka 提交于 
      slab_debug_trace_open() can only be called on caches with SLAB_STORE_USER flag
and as with all slub debugging flags, such caches avoid cpu or percpu partial
slabs altogether, so there's nothing to flush.
Signed-off-by: NVlastimil Babka <vbabka@suse.cz>
Acked-by: NChristoph Lameter <cl@linux.com>
      eafb1d64
  3. 14 8月, 2021 3 次提交
  5. 31 7月, 2021 1 次提交
  7. 18 7月, 2021 1 次提交
  9. 16 7月, 2021 1 次提交
  11. 09 7月, 2021 1 次提交
  13. 30 6月, 2021 5 次提交