Reduce contention on the inode->i_lock by ensuring that we use RCU
when looking up the NFS open context.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Speed up lookups of an existing lock context by avoiding the inode->i_lock,
and using RCU instead.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

For the 'files' and 'flexfiles' layout types, we do not expect the reply
to be any larger than 4k. The block and scsi layout types are a little more
greedy, so we keep allocating the maximum response size for now.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

We don't need a zeroed out array, since it is immediately being filled.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

It is no longer used outside of net/sunrpc/socklib.c
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Simplify the retry logic.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

In preparation for sharing with AF_LOCAL.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Most of this code should also be reusable with other socket types.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Add a bvec array to struct xdr_buf, and have the client allocate it
when we need to receive data into pages.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

If the RPC call relies on the receive call allocating pages as buffers,
then let's label it so that we
a) Don't leak memory by allocating pages for requests that do not expect
   this behaviour
b) Can optimise for the common case where calls do not require allocation.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

We no longer need priority semantics on the xprt->sending queue, because
the order in which tasks are sent is now dictated by their position in
the send queue.
Note that the backlog queue remains a priority queue, meaning that
slot resources are still managed in order of task priority.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Fix up the priority queue to not batch by owner, but by queue, so that
we allow '1 << priority' elements to be dequeued before switching to
the next priority queue.
The owner field is still used to wake up requests in round robin order
by owner to avoid single processes hogging the RPC layer by loading the
queues.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

If the server is slow, we can find ourselves with quite a lot of entries
on the receive queue. Converting the search from an O(n) to O(log(n))
can make a significant difference, particularly since we have to hold
a number of locks while searching.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Treat socket write space handling in the same way we now treat transport
congestion: by denying the XPRT_LOCK until the transport signals that it
has free buffer space.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

The theory was that we would need to grab the socket lock anyway, so we
might as well use it to gate the allocation of RPC slots for a TCP
socket.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

This no longer causes them to lose their place in the transmission queue.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Rather than forcing each and every RPC task to grab the socket write
lock in order to send itself, we allow whichever task is holding the
write lock to attempt to drain the entire transmit queue.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Avoid memory starvation by giving RPCs that are tagged with the
RPC_TASK_SWAPPER flag the highest priority.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Both RDMA and UDP transports require the request to get a "congestion control"
credit before they can be transmitted. Right now, this is done when
the request locks the socket. We'd like it to happen when a request attempts
to be transmitted for the first time.
In order to support retransmission of requests that already hold such
credits, we also want to ensure that they get queued first, so that we
don't deadlock with requests that have yet to obtain a credit.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

One of the intentions with the priority queues was to ensure that no
single process can hog the transport. The field task->tk_owner therefore
identifies the RPC call's origin, and is intended to allow the RPC layer
to organise queues for fairness.
This commit therefore modifies the transmit queue to group requests
by task->tk_owner, and ensures that we round robin among those groups.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Remove the checks for whether or not we need to transmit, and whether
or not a reply has been received. Those are already handled in
call_transmit() itself.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

If the request is still on the queue, this will be incorrect behaviour.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

When we shift to using the transmit queue, then the task that holds the
write lock will not necessarily be the same as the one being transmitted.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Fix up the back channel code to recognise that it has already been
transmitted, so does not need to be called again.
Also ensure that we set req->rq_task.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Move the call encoding so that it occurs before the transport connection
etc.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Add the queue that will enforce the ordering of RPC task transmission.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

When storing a struct rpc_rqst on the slot allocation list, we currently
use the same field 'rq_list' as we use to store the request on the
receive queue. Since the structure is never on both lists at the same
time, this is OK.
However, for clarity, let's make that a union with different names for
the different lists so that we can more easily distinguish between
the two states.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Allow the caller in clnt.c to call into the code to wait for a reply
after calling xprt_transmit(). Again, the reason is that the backchannel
code does not need this functionality.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Separate out the action of adding a request to the reply queue so that the
backchannel code can simply skip calling it altogether.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

We will use the same lock to protect both the transmit and receive queues.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Rather than waking up the entire queue of RPC messages a second time,
just wake up the task that was put to sleep.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

When asked to wake up an RPC task, it makes sense to test whether or not
the task is still queued.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

Add a helper that will wake up a task that is sleeping on a specific
queue, and will set the value of task->tk_status. This is mainly
intended for use by the transport layer to notify the task of an
error condition.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>

We are going to need to pin for both send and receive.
Signed-off-by: NTrond Myklebust <trond.myklebust@hammerspace.com>