• C
    mm: ignore VM_SOFTDIRTY on VMA merging · 34228d47
    Cyrill Gorcunov 提交于
    The VM_SOFTDIRTY bit affects vma merge routine: if two VMAs has all bits
    in vm_flags matched except dirty bit the kernel can't longer merge them
    and this forces the kernel to generate new VMAs instead.
    
    It finally may lead to the situation when userspace application reaches
    vm.max_map_count limit and get crashed in worse case
    
     | (gimp:11768): GLib-ERROR **: gmem.c:110: failed to allocate 4096 bytes
     |
     | (file-tiff-load:12038): LibGimpBase-WARNING **: file-tiff-load: gimp_wire_read(): error
     | xinit: connection to X server lost
     |
     | waiting for X server to shut down
     | /usr/lib64/gimp/2.0/plug-ins/file-tiff-load terminated: Hangup
     | /usr/lib64/gimp/2.0/plug-ins/script-fu terminated: Hangup
     | /usr/lib64/gimp/2.0/plug-ins/script-fu terminated: Hangup
    
      https://bugzilla.kernel.org/show_bug.cgi?id=67651
      https://bugzilla.gnome.org/show_bug.cgi?id=719619#c0
    
    Initial problem came from missed VM_SOFTDIRTY in do_brk() routine but
    even if we would set up VM_SOFTDIRTY here, there is still a way to
    prevent VMAs from merging: one can call
    
     | echo 4 > /proc/$PID/clear_refs
    
    and clear all VM_SOFTDIRTY over all VMAs presented in memory map, then
    new do_brk() will try to extend old VMA and finds that dirty bit doesn't
    match thus new VMA will be generated.
    
    As discussed with Pavel, the right approach should be to ignore
    VM_SOFTDIRTY bit when we're trying to merge VMAs and if merge successed
    we mark extended VMA with dirty bit where needed.
    Signed-off-by: NCyrill Gorcunov <gorcunov@openvz.org>
    Reported-by: NBastian Hougaard <gnome@rvzt.net>
    Reported-by: NMel Gorman <mgorman@suse.de>
    Cc: Pavel Emelyanov <xemul@parallels.com>
    Cc: <stable@vger.kernel.org>
    Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>
    34228d47
mmap.c 86.2 KB