• O
    Keep track of meaningful replication offset in replicas too · 4447ddc8
    Oran Agra 提交于
    Now both master and replicas keep track of the last replication offset
    that contains meaningful data (ignoring the tailing pings), and both
    trim that tail from the replication backlog, and the offset with which
    they try to use for psync.
    
    the implication is that if someone missed some pings, or even have
    excessive pings that the promoted replica has, it'll still be able to
    psync (avoid full sync).
    
    the downside (which was already committed) is that replicas running old
    code may fail to psync, since the promoted replica trims pings form it's
    backlog.
    
    This commit adds a test that reproduces several cases of promotions and
    demotions with stale and non-stale pings
    
    Background:
    The mearningful offset on the master was added recently to solve a problem were
    the master is left all alone, injecting PINGs into it's backlog when no one is
    listening and then gets demoted and tries to replicate from a replica that didn't
    have any of the PINGs (or at least not the last ones).
    
    however, consider this case:
    master A has two replicas (B and C) replicating directly from it.
    there's no traffic at all, and also no network issues, just many pings in the
    tail of the backlog. now B gets promoted, A becomes a replica of B, and C
    remains a replica of A. when A gets demoted, it trims the pings from its
    backlog, and successfully replicate from B. however, C is still aware of
    these PINGs, when it'll disconnect and re-connect to A, it'll ask for something
    that's not in the backlog anymore (since A trimmed the tail of it's backlog),
    and be forced to do a full sync (something it didn't have to do before the
    meaningful offset fix).
    
    Besides that, the psync2 test was always failing randomly here and there, it
    turns out the reason were PINGs. Investigating it shows the following scenario:
    
    cycle 1: redis #1 is master, and all the rest are direct replicas of #1
    cycle 2: redis #2 is promoted to master, #1 is a replica of #2 and #3 is replica of #1
    now we see that when #1 is demoted it prints:
    17339:S 21 Apr 2020 11:16:38.523 * Using the meaningful offset 3929963 instead of 3929977 to exclude the final PINGs (14 bytes difference)
    17339:S 21 Apr 2020 11:16:39.391 * Trying a partial resynchronization (request e2b3f8817735fdfe5fa4626766daa938b61419e5:3929964).
    17339:S 21 Apr 2020 11:16:39.392 * Successful partial resynchronization with master.
    and when #3 connects to the demoted #2, #2 says:
    17339:S 21 Apr 2020 11:16:40.084 * Partial resynchronization not accepted: Requested offset for secondary ID was 3929978, but I can reply up to 3929964
    
    so the issue here is that the meaningful offset feature saved the day for the
    demoted master (since it needs to sync from a replica that didn't get the last
    ping), but it didn't help one of the other replicas which did get the last ping.
    4447ddc8
networking.c 117.6 KB