• B
    sha1_file: introduce close_one_pack() to close packs on fd pressure · 88d0db55
    Brandon Casey 提交于
    When the number of open packs exceeds pack_max_fds, unuse_one_window()
    is called repeatedly to attempt to release the least-recently-used
    pack windows, which, as a side-effect, will also close a pack file
    after closing its last open window.  If a pack file has been opened,
    but no windows have been allocated into it, it will never be selected
    by unuse_one_window() and hence its file descriptor will not be
    closed.  When this happens, git may exceed the number of file
    descriptors permitted by the system.
    
    This latter situation can occur in show-ref or receive-pack during ref
    advertisement.  During ref advertisement, receive-pack will iterate
    over every ref in the repository and advertise it to the client after
    ensuring that the ref exists in the local repository.  If the ref is
    located inside a pack, then the pack is opened to ensure that it
    exists, but since the object is not actually read from the pack, no
    mmap windows are allocated.  When the number of open packs exceeds
    pack_max_fds, unuse_one_window() will not be able to find any windows to
    free and will not be able to close any packs.  Once the per-process
    file descriptor limit is exceeded, receive-pack will produce a warning,
    not an error, for each pack it cannot open, and will then most likely
    fail with an error to spawn rev-list or index-pack like:
    
       error: cannot create standard input pipe for rev-list: Too many open files
       error: Could not run 'git rev-list'
    
    This may also occur during upload-pack when refs are packed (in the
    packed-refs file) and the number of packs that must be opened to
    verify that these packed refs exist exceeds the file descriptor
    limit.  If the refs are loose, then upload-pack will read each ref
    from the object database (if the object is in a pack, allocating one
    or more mmap windows for it) in order to peel tags and advertise the
    underlying object.  But when the refs are packed and peeled,
    upload-pack will use the peeled sha1 in the packed-refs file and
    will not need to read from the pack files, so no mmap windows will
    be allocated and just like with receive-pack, unuse_one_window()
    will never select these opened packs to close.
    
    When we have file descriptor pressure, we just need to find an open
    pack to close.  We can leave the existing mmap windows open.  If
    additional windows need to be mapped into the pack file, it will be
    reopened when necessary.  If the pack file has been rewritten in the
    mean time, open_packed_git_1() should notice when it compares the file
    size or the pack's sha1 checksum to what was previously read from the
    pack index, and reject it.
    
    Let's introduce a new function close_one_pack() designed specifically
    for this purpose to search for and close the least-recently-used pack,
    where LRU is defined as (in order of preference):
    
       * pack with oldest mtime and no allocated mmap windows
       * pack with the least-recently-used windows, i.e. the pack
         with the oldest most-recently-used window, where none of
         the windows are in use
       * pack with the least-recently-used windows
    Signed-off-by: NBrandon Casey <drafnel@gmail.com>
    Signed-off-by: NJunio C Hamano <gitster@pobox.com>
    88d0db55
sha1_file.c 73.4 KB