If client stream does not have any data to sink and neither received
EOF, a dummy packet is sent to the daemon signalising client is ready to
sink some data. However, after we added event loop to client a race may
occur:

Thread 1 calls virNetClientStreamRecvPacket and since no data are cached
nor stream has EOF, it decides to send dummy packet to server which will
sent some data in turn. However, during this decision and actual message
exchange with server -

Thread 2 receives last stream data from server. Therefore an EOF is set
on stream and if there is a call waiting (which is not yet) it is woken
up. However, Thread 1 haven't sent anything so far, so there is no call
to be woken up. So this thread sent dummy packet to daemon, which
ignores that as no stream is associated with such packet and therefore
no reply will ever come.

This race causes client to hang indefinitely.

The RPC code had several latent memory leaks and an attempt to
free the wrong string, but thankfully nothing triggered them
(blkiotune was the only one returning a string, and always as
the last parameter).  Also, our cleanups for rpcgen ended up
nuking a line of code that renders VIR_TYPED_PARAM_INT broken,
because it was the only use of 'i' in a function, even though
it was a member usage rather than a standalone declaration.

* daemon/remote.c (remoteSerializeTypedParameters): Free the
correct array element.
(remoteDispatchDomainGetSchedulerParameters)
(remoteDispatchDomainGetSchedulerParametersFlags)
(remoteDispatchDomainBlockStatsFlags)
(remoteDispatchDomainGetMemoryParameters): Don't leak strings.
* src/rpc/genprotocol.pl: Don't nuke member-usage of 'buf' or 'i'.

When one thread passes the buck to another thread, it uses
virCondSignal to wake up the target thread. The variable
'haveTheBuck' is not updated in a race-free manner when
this occurs. The current thread sets it to false, and the
woken up thread sets it to true. There is a window where
a 3rd thread can come in and grab the buck.

Even if this didn't lead to crashes & deadlocks, this would
still result in unfairness in the buckpassing algorithm.

A better solution is to *never* set haveTheBuck to false
when we're passing the buck. Only set it to false when there
is no further thread waiting for the buck.

* src/rpc/virnetclient.c: Only set haveTheBuck to false
  if no thread is waiting

Commit fd066925 tried to fix
a race condition in

  commit fa959500
  Author: Daniel P. Berrange <berrange@redhat.com>
  Date:   Fri Nov 11 15:28:41 2011 +0000

    Explicitly track whether the buck is held in remote client

Unfortunately there is a second race condition whereby the
event loop can trigger due to incoming data to read. Revert
this fix, so a complete fix for the problem can be cleanly
applied

* src/rpc/virnetclient.c: Revert fd066925

Currently if you try to connect to a local libvirtd when
libvirtd is not in $PATH, you'll get an error

  error: internal error invalid use of command API

This is because remoteFindDaemonPath() returns NULL, which
causes us to pass NULL into virNetSocketConnectUNIX which
in turn causes us to pass NULL into virCommandNewArgList.

Adding missing error checks improves this to

  error: internal error Unable to locate libvirtd daemon in $PATH

* src/remote/remote_driver.c: Report error if libvirtd
  cannot be found
* src/rpc/virnetsocket.c: Report error if caller requested
  spawning of daemon, but provided no binary path

https://bugzilla.redhat.com/show_bug.cgi?id=648855 mentioned a
misuse of 'an' where 'a' is proper; that has since been fixed,
but a search found other problems (some were a spelling error for
'and', while most were fixed by 'a').

* daemon/stream.c: Fix grammar.
* src/conf/domain_conf.c: Likewise.
* src/conf/domain_event.c: Likewise.
* src/esx/esx_driver.c: Likewise.
* src/esx/esx_vi.c: Likewise.
* src/rpc/virnetclient.c: Likewise.
* src/rpc/virnetserverprogram.c: Likewise.
* src/storage/storage_backend_fs.c: Likewise.
* src/util/conf.c: Likewise.
* src/util/dnsmasq.c: Likewise.
* src/util/iptables.c: Likewise.
* src/xen/xen_hypervisor.c: Likewise.
* src/xen/xend_internal.c: Likewise.
* src/xen/xs_internal.c: Likewise.
* tools/virsh.c: Likewise.

The RPC fixups needed on Linux are also needed on cygwin, and
worked without further tweaking to the list of fixups.  Also,
unlike BSD, Cygwin exports 'struct ifreq', but unlike Linux,
Cygwin lacks the ioctls that we were using 'struct ifreq' to
access.  This patch allows compilation under cygwin.

* src/rpc/genprotocol.pl: Also perform fixups on cygwin.
* src/util/virnetdev.c (HAVE_STRUCT_IFREQ): Also require AF_PACKET
definition.
* src/util/virnetdevbridge.c (virNetDevSetupControlFull): Only
compile if SIOCBRADDBR works.

Originaly, the code checked if another client is the queue and infered
ownership of the buck from that. Commit fa959500
added a separate variable to track the buck. That caused, that a new
call might enter claiming it has the buck, while another thread was
signalled to take the buck. This ends in two threads claiming they hold
the buck and entering poll(). This happens due to a race on waking up
threads on the client lock mutex.

This caused multi-threaded clients to hang, most prominently visible and
reproducible on python based clients, like virt-manager.

This patch causes threads, that have been signalled to take the buck to
re-check if buck is held by another thread.

Detected by Coverity. Leak introduced in commit 673adba5.

Two separate bugs here:
1. call was not freed on all error paths
2. virCondDestroy was called even if virCondInit failed
Signed-off-by: NAlex Jia <ajia@redhat.com>
Signed-off-by: NEric Blake <eblake@redhat.com>

When another thread was dispatching while we wanted to send a
non-blocking call, we correctly queued the call and woke up the thread
but the thread just threw the call away since it forgot to recheck if
its socket was writable.

When spawning an ssh connection, the environment variables
DISPLAY, SSH_ASKPASS, ... are passed. However XAUTHORITY,
which is necessary if the .Xauthority is in a non default
place, was not passed.
Signed-off-by: NChristian Franke <nobody@nowhere.ws>

These APIs are used by both client and server RPC layer to handle
processing of keepalive messages.

The keepalive program has two procedures: PING, and PONG.
Both are used only in asynchronous messages and the sender doesn't wait
for any reply. However, the party which receives PING messages is
supposed to react by sending PONG message the other party, but no
explicit binding between PING and PONG messages is made. For backward
compatibility neither server nor client are allowed to send keepalive
messages before checking that remote party supports them.

When virNetClientIOEventLoop is called for a non-blocking call and not
even a single byte can be sent from this call without blocking, we
properly reported that to the caller which properly frees the call. But
we never removed the call from a call queue.

Due to the asynchronous nature of streams, we might continue to
receive some stream packets from the server even after we have
shutdown the stream on the client side. These should be discarded
silently, rather than raising an error in the RPC layer.

* src/rpc/virnetclient.c: Discard stream data silently

Add a new virNetClientSendNonBlock which returns 2 on
full send, 1 on partial send, 0 on no send, -1 on error

If a partial send occurs, then a subsequent call to any
of the virNetClientSend* APIs will finish any outstanding
I/O.

TODO: the virNetClientEvent event handler could be used
to speed up completion of partial sends if an event loop
is present.

* src/rpc/virnetsocket.h, src/rpc/virnetsocket.c: Add new
  virNetSocketHasPendingData() API to test for cached
  data pending send.
* src/rpc/virnetclient.c, src/rpc/virnetclient.h: Add new
  virNetClientSendNonBlock() API to send non-blocking API

* src/rpc/virnetclient.c: Add helper for setting I/O callback events

Stop multiplexing virNetClientSend for two different purposes,
instead add virNetClientSendWithReply and virNetClientSendNoReply

* src/rpc/virnetclient.c, src/rpc/virnetclient.h: Replace
  virNetClientSend with virNetClientSendWithReply and
  virNetClientSendNoReply
* src/rpc/virnetclientprogram.c, src/rpc/virnetclientstream.c:
  Update for new API names

Remove some duplication by pulling the code for passing the
buck out into a helper method

* src/rpc/virnetclient.c: Introduce virNetClientIOEventLoopPassTheBuck

Instead of inferring whether the buck is held from the waitDispatch
pointer, use an explicit 'bool haveTheBuck' field

* src/rpc/virnetclient.c: Explicitly track the buck

Directly messing around with the linked list is potentially
dangerous. Introduce some helper APIs to deal with list
manipulating the list

* src/rpc/virnetclient.c: Create linked list handlers

The src/util/network.c file is a dumping ground for many different
APIs. Split it up into 5 pieces, along functional lines

 - src/util/virnetdevbandwidth.c: virNetDevBandwidth type & helper APIs
 - src/util/virnetdevvportprofile.c: virNetDevVPortProfile type & helper APIs
 - src/util/virsocketaddr.c: virSocketAddr and APIs
 - src/conf/netdev_bandwidth_conf.c: XML parsing / formatting
   for virNetDevBandwidth
 - src/conf/netdev_vport_profile_conf.c: XML parsing / formatting
   for virNetDevVPortProfile

* src/util/network.c, src/util/network.h: Split into 5 pieces
* src/conf/netdev_bandwidth_conf.c, src/conf/netdev_bandwidth_conf.h,
  src/conf/netdev_vport_profile_conf.c, src/conf/netdev_vport_profile_conf.h,
  src/util/virnetdevbandwidth.c, src/util/virnetdevbandwidth.h,
  src/util/virnetdevvportprofile.c, src/util/virnetdevvportprofile.h,
  src/util/virsocketaddr.c, src/util/virsocketaddr.h: New pieces
* daemon/libvirtd.h, daemon/remote.c, src/conf/domain_conf.c,
  src/conf/domain_conf.h, src/conf/network_conf.c,
  src/conf/network_conf.h, src/conf/nwfilter_conf.h,
  src/esx/esx_util.h, src/network/bridge_driver.c,
  src/qemu/qemu_conf.c, src/rpc/virnetsocket.c,
  src/rpc/virnetsocket.h, src/util/dnsmasq.h, src/util/interface.h,
  src/util/iptables.h, src/util/macvtap.c, src/util/macvtap.h,
  src/util/virnetdev.h, src/util/virnetdevtap.c,
  tools/virsh.c: Update include files

Send and receive string typed parameters across RPC.  This also
completes the back-compat mentioned in the previous patch - the
only time we have an older client talking to a newer server is
if RPC is in use, so filtering out strings during RPC prevents
returning an unknown type to the older client.

* src/remote/remote_protocol.x (remote_typed_param_value): Add
another union value.
* daemon/remote.c (remoteDeserializeTypedParameters): Handle
strings on rpc.
(remoteSerializeTypedParameters): Likewise; plus filter out
strings when replying to older clients.  Adjust callers.
* src/remote/remote_driver.c (remoteFreeTypedParameters)
(remoteSerializeTypedParameters)
(remoteDeserializeTypedParameters): Handle strings on rpc.
* src/rpc/gendispatch.pl: Properly clean up typed arrays.
* src/remote_protocol-structs: Update.
Based on an initial patch by Hu Tao, with feedback from
Daniel P. Berrange.
Signed-off-by: NEric Blake <eblake@redhat.com>

The socket address APIs in src/util/network.h either take the
form  virSocketAddrXXX, virSocketXXX or virSocketXXXAddr.

Sanitize this so everything is virSocketAddrXXXX, and ensure
that the virSocketAddr parameter is always the first one.

* src/util/network.c, src/util/network.h: Santize socket
  address API naming
* src/conf/domain_conf.c, src/conf/network_conf.c,
  src/conf/nwfilter_conf.c, src/network/bridge_driver.c,
  src/nwfilter/nwfilter_ebiptables_driver.c,
  src/nwfilter/nwfilter_learnipaddr.c,
  src/qemu/qemu_command.c, src/rpc/virnetsocket.c,
  src/util/dnsmasq.c, src/util/iptables.c,
  src/util/virnetdev.c, src/vbox/vbox_tmpl.c: Update for
  API renaming

The code calling sendfd/recvfd was mistakenly assuming those
calls would never block. They can in fact return EAGAIN and
this is causing us to drop the client connection when blocking
ocurrs while sending/receiving FDs.

Fixing this is a little hairy on the incoming side, since at
the point where we see the EAGAIN, we already thought we had
finished receiving all data for the packet. So we play a little
trick to reset bufferOffset again and go back into polling for
more data.

* src/rpc/virnetsocket.c, src/rpc/virnetsocket.h: Update
  virNetSocketSendFD/RecvFD to return 0 on EAGAIN, or 1
  on success
* src/rpc/virnetclient.c: Move decoding of header & fds
  out of virNetClientCallDispatch and into virNetClientIOHandleInput.
  Handling blocking when sending/receiving FDs
* src/rpc/virnetmessage.h: Add a 'donefds' field to track
  how many FDs we've sent / received
* src/rpc/virnetserverclient.c: Handling blocking when
  sending/receiving FDs

I ran into the following build failure:
$ mkdir -p build1 build2/a/very/deep/hierarcy
$ cd build2/a/very/deep/hierarcy
$ ../../../../../configure && make
$ cd ../../../../build1
$ ../configure && make
...
../../src/remote/remote_protocol.c:7:55: fatal error: ../../../../../src/remote/remote_protocol.h: No such file or directory

Turns out that we were sometimes generating the remote_protocol.c
file with information from the VPATH build, which is bad, since
any file shipped in the tarball should be idempotent no matter how
deep the VPATH build tree that created it.

* src/rpc/genprotocol.pl: Don't embed VPATH into generated file.

If daemon is using SASL it reads client data into a cache. This cache is
big (usually 65KB) and can thus contain 2 or more messages. However,
on socket event we can dispatch only one message. So if we read two
messages at once, the second will not be dispatched as the socket event
goes away with filling the cache.
Moreover, when dispatching the cache we need to remember to take care
of client max requests limit.

The RPC server classes are extended to allow FDs to be received
from clients with calls. There is not currently any way for a
procedure to pass FDs back to the client with replies

* daemon/remote.c, src/rpc/gendispatch.pl: Change virNetMessageHeaderPtr
  param to virNetMessagePtr in dispatcher impls
* src/rpc/virnetserver.c, src/rpc/virnetserverclient.c,
  src/rpc/virnetserverprogram.c, src/rpc/virnetserverprogram.h:
  Extend to support FD passing

Extend the RPC client code to allow file descriptors to be sent
to the server with calls, and received back with replies.

* src/remote/remote_driver.c: Stub extra args
* src/libvirt_private.syms, src/rpc/virnetclient.c,
  src/rpc/virnetclient.h, src/rpc/virnetclientprogram.c,
  src/rpc/virnetclientprogram.h: Extend APIs to allow
  FD passing

Define two new RPC message types VIR_NET_CALL_WITH_FDS and
VIR_NET_REPLY_WITH_FDS. These message types are equivalent
to VIR_NET_CALL and VIR_NET_REPLY, except that between the
message header, and payload there is a 32-bit integer field
specifying how many file descriptors have been passed.

The actual file descriptors are sent/recv'd out of band.

* src/rpc/virnetmessage.c, src/rpc/virnetmessage.h,
  src/libvirt_private.syms: Add support for handling
  passed file descriptors
* src/rpc/virnetprotocol.x: Extend protocol for FD
  passing

Add APIs to the virNetSocket object, to allow file descriptors
to be sent/received over UNIX domain socket connections

* src/rpc/virnetsocket.c, src/rpc/virnetsocket.h,
  src/libvirt_private.syms: Add APIs for FD send/recv

to escape the netcat command since it's passed to the shell. Adjust
expected test case output accordingly.