• A
    libceph: have messages take a connection reference · 92ce034b
    Alex Elder 提交于
    There are essentially two types of ceph messages: incoming and
    outgoing.  Outgoing messages are always allocated via ceph_msg_new(),
    and at the time of their allocation they are not associated with any
    particular connection.  Incoming messages are always allocated via
    ceph_con_in_msg_alloc(), and they are initially associated with the
    connection from which incoming data will be placed into the message.
    
    When an outgoing message gets sent, it becomes associated with a
    connection and remains that way until the message is successfully
    sent.  The association of an incoming message goes away at the point
    it is sent to an upper layer via a con->ops->dispatch method.
    
    This patch implements reference counting for all ceph messages, such
    that every message holds a reference (and a pointer) to a connection
    if and only if it is associated with that connection (as described
    above).
    
    
    For background, here is an explanation of the ceph message
    lifecycle, emphasizing when an association exists between a message
    and a connection.
    
    Outgoing Messages
    An outgoing message is "owned" by its allocator, from the time it is
    allocated in ceph_msg_new() up to the point it gets queued for
    sending in ceph_con_send().  Prior to that point the message's
    msg->con pointer is null; at the point it is queued for sending its
    message pointer is assigned to refer to the connection.  At that
    time the message is inserted into a connection's out_queue list.
    
    When a message on the out_queue list has been sent to the socket
    layer to be put on the wire, it is transferred out of that list and
    into the connection's out_sent list.  At that point it is still owned
    by the connection, and will remain so until an acknowledgement is
    received from the recipient that indicates the message was
    successfully transferred.  When such an acknowledgement is received
    (in process_ack()), the message is removed from its list (in
    ceph_msg_remove()), at which point it is no longer associated with
    the connection.
    
    So basically, any time a message is on one of a connection's lists,
    it is associated with that connection.  Reference counting outgoing
    messages can thus be done at the points a message is added to the
    out_queue (in ceph_con_send()) and the point it is removed from
    either its two lists (in ceph_msg_remove())--at which point its
    connection pointer becomes null.
    
    Incoming Messages
    When an incoming message on a connection is getting read (in
    read_partial_message()) and there is no message in con->in_msg,
    a new one is allocated using ceph_con_in_msg_alloc().  At that
    point the message is associated with the connection.  Once that
    message has been completely and successfully read, it is passed to
    upper layer code using the connection's con->ops->dispatch method.
    At that point the association between the message and the connection
    no longer exists.
    
    Reference counting of connections for incoming messages can be done
    by taking a reference to the connection when the message gets
    allocated, and releasing that reference when it gets handed off
    using the dispatch method.
    
    We should never fail to get a connection reference for a
    message--the since the caller should already hold one.
    Signed-off-by: NAlex Elder <elder@inktank.com>
    Reviewed-by: NSage Weil <sage@inktank.com>
    92ce034b
messenger.c 66.5 KB