Directly messing around with the linked list is potentially
dangerous. Introduce some helper APIs to deal with list
manipulating the list

* src/rpc/virnetclient.c: Create linked list handlers

The src/util/network.c file is a dumping ground for many different
APIs. Split it up into 5 pieces, along functional lines

 - src/util/virnetdevbandwidth.c: virNetDevBandwidth type & helper APIs
 - src/util/virnetdevvportprofile.c: virNetDevVPortProfile type & helper APIs
 - src/util/virsocketaddr.c: virSocketAddr and APIs
 - src/conf/netdev_bandwidth_conf.c: XML parsing / formatting
   for virNetDevBandwidth
 - src/conf/netdev_vport_profile_conf.c: XML parsing / formatting
   for virNetDevVPortProfile

* src/util/network.c, src/util/network.h: Split into 5 pieces
* src/conf/netdev_bandwidth_conf.c, src/conf/netdev_bandwidth_conf.h,
  src/conf/netdev_vport_profile_conf.c, src/conf/netdev_vport_profile_conf.h,
  src/util/virnetdevbandwidth.c, src/util/virnetdevbandwidth.h,
  src/util/virnetdevvportprofile.c, src/util/virnetdevvportprofile.h,
  src/util/virsocketaddr.c, src/util/virsocketaddr.h: New pieces
* daemon/libvirtd.h, daemon/remote.c, src/conf/domain_conf.c,
  src/conf/domain_conf.h, src/conf/network_conf.c,
  src/conf/network_conf.h, src/conf/nwfilter_conf.h,
  src/esx/esx_util.h, src/network/bridge_driver.c,
  src/qemu/qemu_conf.c, src/rpc/virnetsocket.c,
  src/rpc/virnetsocket.h, src/util/dnsmasq.h, src/util/interface.h,
  src/util/iptables.h, src/util/macvtap.c, src/util/macvtap.h,
  src/util/virnetdev.h, src/util/virnetdevtap.c,
  tools/virsh.c: Update include files

Send and receive string typed parameters across RPC.  This also
completes the back-compat mentioned in the previous patch - the
only time we have an older client talking to a newer server is
if RPC is in use, so filtering out strings during RPC prevents
returning an unknown type to the older client.

* src/remote/remote_protocol.x (remote_typed_param_value): Add
another union value.
* daemon/remote.c (remoteDeserializeTypedParameters): Handle
strings on rpc.
(remoteSerializeTypedParameters): Likewise; plus filter out
strings when replying to older clients.  Adjust callers.
* src/remote/remote_driver.c (remoteFreeTypedParameters)
(remoteSerializeTypedParameters)
(remoteDeserializeTypedParameters): Handle strings on rpc.
* src/rpc/gendispatch.pl: Properly clean up typed arrays.
* src/remote_protocol-structs: Update.
Based on an initial patch by Hu Tao, with feedback from
Daniel P. Berrange.
Signed-off-by: NEric Blake <eblake@redhat.com>

The socket address APIs in src/util/network.h either take the
form  virSocketAddrXXX, virSocketXXX or virSocketXXXAddr.

Sanitize this so everything is virSocketAddrXXXX, and ensure
that the virSocketAddr parameter is always the first one.

* src/util/network.c, src/util/network.h: Santize socket
  address API naming
* src/conf/domain_conf.c, src/conf/network_conf.c,
  src/conf/nwfilter_conf.c, src/network/bridge_driver.c,
  src/nwfilter/nwfilter_ebiptables_driver.c,
  src/nwfilter/nwfilter_learnipaddr.c,
  src/qemu/qemu_command.c, src/rpc/virnetsocket.c,
  src/util/dnsmasq.c, src/util/iptables.c,
  src/util/virnetdev.c, src/vbox/vbox_tmpl.c: Update for
  API renaming

The code calling sendfd/recvfd was mistakenly assuming those
calls would never block. They can in fact return EAGAIN and
this is causing us to drop the client connection when blocking
ocurrs while sending/receiving FDs.

Fixing this is a little hairy on the incoming side, since at
the point where we see the EAGAIN, we already thought we had
finished receiving all data for the packet. So we play a little
trick to reset bufferOffset again and go back into polling for
more data.

* src/rpc/virnetsocket.c, src/rpc/virnetsocket.h: Update
  virNetSocketSendFD/RecvFD to return 0 on EAGAIN, or 1
  on success
* src/rpc/virnetclient.c: Move decoding of header & fds
  out of virNetClientCallDispatch and into virNetClientIOHandleInput.
  Handling blocking when sending/receiving FDs
* src/rpc/virnetmessage.h: Add a 'donefds' field to track
  how many FDs we've sent / received
* src/rpc/virnetserverclient.c: Handling blocking when
  sending/receiving FDs

I ran into the following build failure:
$ mkdir -p build1 build2/a/very/deep/hierarcy
$ cd build2/a/very/deep/hierarcy
$ ../../../../../configure && make
$ cd ../../../../build1
$ ../configure && make
...
../../src/remote/remote_protocol.c:7:55: fatal error: ../../../../../src/remote/remote_protocol.h: No such file or directory

Turns out that we were sometimes generating the remote_protocol.c
file with information from the VPATH build, which is bad, since
any file shipped in the tarball should be idempotent no matter how
deep the VPATH build tree that created it.

* src/rpc/genprotocol.pl: Don't embed VPATH into generated file.

If daemon is using SASL it reads client data into a cache. This cache is
big (usually 65KB) and can thus contain 2 or more messages. However,
on socket event we can dispatch only one message. So if we read two
messages at once, the second will not be dispatched as the socket event
goes away with filling the cache.
Moreover, when dispatching the cache we need to remember to take care
of client max requests limit.

The RPC server classes are extended to allow FDs to be received
from clients with calls. There is not currently any way for a
procedure to pass FDs back to the client with replies

* daemon/remote.c, src/rpc/gendispatch.pl: Change virNetMessageHeaderPtr
  param to virNetMessagePtr in dispatcher impls
* src/rpc/virnetserver.c, src/rpc/virnetserverclient.c,
  src/rpc/virnetserverprogram.c, src/rpc/virnetserverprogram.h:
  Extend to support FD passing

Extend the RPC client code to allow file descriptors to be sent
to the server with calls, and received back with replies.

* src/remote/remote_driver.c: Stub extra args
* src/libvirt_private.syms, src/rpc/virnetclient.c,
  src/rpc/virnetclient.h, src/rpc/virnetclientprogram.c,
  src/rpc/virnetclientprogram.h: Extend APIs to allow
  FD passing

Define two new RPC message types VIR_NET_CALL_WITH_FDS and
VIR_NET_REPLY_WITH_FDS. These message types are equivalent
to VIR_NET_CALL and VIR_NET_REPLY, except that between the
message header, and payload there is a 32-bit integer field
specifying how many file descriptors have been passed.

The actual file descriptors are sent/recv'd out of band.

* src/rpc/virnetmessage.c, src/rpc/virnetmessage.h,
  src/libvirt_private.syms: Add support for handling
  passed file descriptors
* src/rpc/virnetprotocol.x: Extend protocol for FD
  passing

Add APIs to the virNetSocket object, to allow file descriptors
to be sent/received over UNIX domain socket connections

* src/rpc/virnetsocket.c, src/rpc/virnetsocket.h,
  src/libvirt_private.syms: Add APIs for FD send/recv

to escape the netcat command since it's passed to the shell. Adjust
expected test case output accordingly.

Based on a patch by Marc Deslauriers <marc.deslauriers@ubuntu.com>

RH: https://bugzilla.redhat.com/show_bug.cgi?id=562176
Ubuntu: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/517478
Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=573172

The libvirtd daemon had a few crude system tap probes. Some of
these were broken during the RPC rewrite. The new modular RPC
code is structured in a way that allows much more effective
tracing. Instead of trying to hook up the original probes,
define a new set of probes for the RPC and event code.

The master probes file is now src/probes.d.  This contains
probes for virNetServerClientPtr, virNetClientPtr, virSocketPtr
virNetTLSContextPtr and virNetTLSSessionPtr modules. Also add
probes for the poll event loop.

The src/dtrace2systemtap.pl script can convert the probes.d
file into a libvirt_probes.stp file to make use from systemtap
much simpler.

The src/rpc/gensystemtap.pl script can generate a set of
systemtap functions for translating RPC enum values into
printable strings. This works for all RPC header enums (program,
type, status, procedure) and also the authentication enum

The PROBE macro will automatically generate a VIR_DEBUG
statement, so any place with a PROBE can remove any existing
manual DEBUG statements.

* daemon/libvirtd.stp, daemon/probes.d: Remove obsolete probing
* daemon/libvirtd.h: Remove probe macros
* daemon/Makefile.am: Remove all probe buildings/install
* daemon/remote.c: Update authentication probes
* src/dtrace2systemtap.pl, src/rpc/gensystemtap.pl: Scripts
  to generate STP files
* src/internal.h: Add probe macros
* src/probes.d: Master list of probes
* src/rpc/virnetclient.c, src/rpc/virnetserverclient.c,
  src/rpc/virnetsocket.c, src/rpc/virnettlscontext.c,
  src/util/event_poll.c: Insert probe points, removing any
  DEBUG statements that duplicate the info

The virNetTLSContextRef API forgot to acquire/release the lock
while changing ctxt->refs

* src/rpc/virnettlscontext.c: Add lock calls

Pull the call to gnutls_x509_crt_get_dn up into a higher function
so that the 'dname' variable will be available for probe points

* src/rpc/virnettlscontext.c: Pull gnutls_x509_crt_get_dn up
  one level

Instead of directly manipulating sock->refs, add a virSocketRef
API

* src/rpc/virnetsocket.c, src/rpc/virnetsocket.h: Add virSocketRef

If we receive an error on the stream, set the EOF marker so
that any further (bogus) incoming data is dropped.

* src/rpc/virnetclientstream.c: Set EOF on stream

If we send back an unknown program error for async messages,
we will confuse the client because they only expect replies
for method calls. Just log & drop any invalid async messages

* src/rpc/virnetserver.c: Don't send error for async messages

Commit 597fe3ce accidentally
introduced a deadlock when reporting an unknown RPC program.
The virNetServerDispatchNewMessage method is called with
the client locked, and must therefore not attempt to send
any RPC messages back to the client. Only once the incoming
message is passed off to the virNetServerHandleJob worker
is it safe to start sending messages back

* src/rpc/virnetserver.c: Delay checking for unknown RPC
  program until in worker thread

Do not crash if virStreamFinish is called after error.

==11000== Invalid read of size 4
==11000==    at 0x373A8099A0: pthread_mutex_lock (pthread_mutex_lock.c:51)
==11000==    by 0x4C7CADE: virMutexLock (threads-pthread.c:85)
==11000==    by 0x4D57C31: virNetClientStreamRaiseError (virnetclientstream.c:203)
==11000==    by 0x4D385E4: remoteStreamFinish (remote_driver.c:3541)
==11000==    by 0x4D182F9: virStreamFinish (libvirt.c:14157)
==11000==    by 0x40FDC4: cmdScreenshot (virsh.c:3075)
==11000==    by 0x42BA40: vshCommandRun (virsh.c:14922)
==11000==    by 0x42ECCA: main (virsh.c:16381)
==11000==  Address 0x59b86c0 is 16 bytes inside a block of size 216 free'd
==11000==    at 0x4A06928: free (vg_replace_malloc.c:427)
==11000==    by 0x4C69E2B: virFree (memory.c:310)
==11000==    by 0x4D57B56: virNetClientStreamFree (virnetclientstream.c:184)
==11000==    by 0x4D3DB7A: remoteDomainScreenshot (remote_client_bodies.h:1812)
==11000==    by 0x4CFD245: virDomainScreenshot (libvirt.c:2903)
==11000==    by 0x40FB73: cmdScreenshot (virsh.c:3029)
==11000==    by 0x42BA40: vshCommandRun (virsh.c:14922)
==11000==    by 0x42ECCA: main (virsh.c:16381)

Mostly straight-forward, although this is the first API that
returns a new snapshot based on a snapshot rather than a domain.

* src/remote/remote_protocol.x
(REMOTE_PROC_DOMAIN_SNAPSHOT_GET_PARENT): New rpc.
(remote_domain_snapshot_get_parent_args)
(remote_domain_snapshot_get_parent_ret): New structs.
* src/rpc/gendispatch.pl: Adjust generator.
* src/remote/remote_driver.c (remote_driver): Use it.
* src/remote_protocol-structs: Update.

as they are not used with debugging turned off.

commit 984840a2 removed the
notification of waiting calls when VIR_NET_CONTINUE messages
arrive. This was to fix the case of a virStreamAbort() call
being prematurely notified of completion.

The problem is that sometimes there are dummy calls from a
virStreamRecv() call waiting that *do* need to be notified.

These dummy calls should have a status VIR_NET_CONTINUE. So
re-add the notification upon VIR_NET_CONTINUE, but only if
the waiter also has a status of VIR_NET_CONTINUE.

* src/rpc/virnetclient.c: Notify waiting call if stream data
  arrives
* src/rpc/virnetclientstream.c:  Mark dummy stream read packet
  with status VIR_NET_CONTINUE

Libvirt special-cases a specific VIR_ERR_RPC from the remote driver
back into VIR_ERR_NO_SUPPORT on the client, so that clients can
handle missing rpc functions the same whether the hypervisor driver
is local or remote.  However, commit c1b22644 introduced a regression:
VIR_FROM_THIS changed from VIR_FROM_REMOTE to VIR_FROM_RPC, so the
special casing no longer works if the server uses the newer error
domain.

* src/rpc/virnetclientprogram.c
(virNetClientProgramDispatchError): Also cater to 0.9.3 and newer.

So we can allow GSSAPI authentication for ssh.
Signed-off-by: NMatthias Witte <witte@netzquadrat.de>

* src/rpc/virnettlscontext.c: fix memory leak on
  virNetTLSContextValidCertificate.

* Detected in valgrind run:

==25667==
==25667== 6,085 (44 direct, 6,041 indirect) bytes in 1 blocks are definitely
lost in loss record 326 of 351
==25667==    at 0x4005447: calloc (vg_replace_malloc.c:467)
==25667==    by 0x4F2791F3: _asn1_add_node_only (structure.c:53)
==25667==    by 0x4F27997A: _asn1_copy_structure3 (structure.c:421)
==25667==    by 0x4F276A50: _asn1_append_sequence_set (element.c:144)
==25667==    by 0x4F2743FF: asn1_der_decoding (decoding.c:1194)
==25667==    by 0x4F22B9CC: gnutls_x509_crt_import (x509.c:229)
==25667==    by 0x805274B: virNetTLSContextCheckCertificate
(virnettlscontext.c:1009)
==25667==    by 0x804DE32: testTLSSessionInit (virnettlscontexttest.c:693)
==25667==    by 0x804F14D: virtTestRun (testutils.c:140)
==25667==
==25667== 23,188 (88 direct, 23,100 indirect) bytes in 11 blocks are definitely
lost in loss record 346 of 351
==25667==    at 0x4005447: calloc (vg_replace_malloc.c:467)
==25667==    by 0x4F22B841: gnutls_x509_crt_init (x509.c:50)
==25667==    by 0x805272B: virNetTLSContextCheckCertificate
(virnettlscontext.c:1003)
==25667==    by 0x804DDD1: testTLSSessionInit (virnettlscontexttest.c:673)
==25667==    by 0x804F14D: virtTestRun (testutils.c:140)

* How to reproduce?
% cd libvirt && ./configure && make && make -C tests valgrind
or
% valgrind -v --leak-check=full ./tests/virnettlscontexttest
Signed-off-by: NAlex Jia <ajia@redhat.com>

This patch annotates APIs with low or high priority.
In low set MUST be all APIs which might eventually access monitor
(and thus block indefinitely). Other APIs may be marked as high
priority. However, some must be (e.g. domainDestroy).

For high priority calls (HPC), there are some high priority workers
(HPW) created in the pool. HPW can execute only HPC, although normal
worker can process any call regardless priority. Therefore, only those
APIs which are guaranteed to end in reasonable small amount of time
can be marked as HPC.

The size of this HPC pool is static, because HPC are expected to end
quickly, therefore jobs assigned to this pool will be served quickly.
It can be configured in libvirtd.conf via prio_workers variable.
Default is set to 5.

To mark API with low or high priority, append priority:{low|high} to
it's comment in src/remote/remote_protocol.x. This is similar to
autogen|skipgen. If not marked, the generator assumes low as default.

Includes impl of python binding since the generator was not
able to cope.

Note: Requires gendispatch.pl patch from Matthias Bolte

https://www.redhat.com/archives/libvir-list/2011-August/msg01367.html

Commit 2c85644b attempted to
fix a problem with tracking RPC messages from streams by doing

-            if (msg->header.type == VIR_NET_REPLY) {
+            if (msg->header.type == VIR_NET_REPLY ||
+                (msg->header.type == VIR_NET_STREAM &&
+                 msg->header.status != VIR_NET_CONTINUE)) {
                 client->nrequests--;

In other words any stream packet, with status NET_OK or NET_ERROR
would cause nrequests to be decremented. This is great if the
packet from from a synchronous virStreamFinish or virStreamAbort
API call, but wildly wrong if from a server initiated abort.
The latter resulted in 'nrequests' being decremented below zero.
This then causes all I/O for that client to be stopped.

Instead of trying to infer whether we need to decrement the
nrequests field, from the message type/status, introduce an
explicit 'bool tracked' field to mark whether the virNetMessagePtr
object is subject to tracking.

Also add a virNetMessageClear function to allow a message
contents to be cleared out, without adversely impacting the
'tracked' field as a naive memset() would do

* src/rpc/virnetmessage.c, src/rpc/virnetmessage.h: Add
  a 'bool tracked' field and virNetMessageClear() API
* daemon/remote.c, daemon/stream.c, src/rpc/virnetclientprogram.c,
  src/rpc/virnetclientstream.c, src/rpc/virnetserverclient.c,
  src/rpc/virnetserverprogram.c: Switch over to use
  virNetMessageClear() and pass in the 'bool tracked' value
  when creating messages.

The bufferOffset has been initialized to zero in virNetMessageEncodePayloadRaw(),
so, we use bufferLength to represent the length of message which is going to be
sent to client side.

From: Matthias Bolte <matthias.bolte@googlemail.com>
Tested-by: NJim Fehlig <jfehlig@novell.com>

Matthias provided this patch to fix an issue I encountered in the
generator with APIs containing call-by-ref long type, e.g.

int virDomainMigrateGetMaxSpeed(virDomainPtr domain,
                                unsigned long *bandwidth,
                                unsigned int flags);

In case we add a new program in the future (we did that in the past and
we are going to do it again soon) current daemon will behave badly with
new client that wants to use the new program. Before the RPC rewrite we
used to just send an error reply to any request with unknown program.
With the RPC rewrite in 0.9.3 the daemon just closes the connection
through which such request was sent. This patch fixes this regression.

My previous patch 74c75671
introduced a regression by removing TLS initialization from client.

When spice_tls is set but listen_tls is not, we don't initialize
GnuTLS library. So any later gnutls call (e.g. during migration,
where we initialize a certificate) will access uninitialized GnuTLS
internal structs and throws an error.

Although, we might now initialize GnuTLS twice, it is safe according
to the documentation:

    This function can be called many times,
    but will only do something the first time.

This patch creates 2 functions: virNetTLSInit and virNetTLSDeinit
with respect to written above.

Regression introduced in commit 5d30db09.

* src/rpc/virnetsocket.c (virNetSocketNewListenUNIX) [WIN32]: Use
correct signature.

If a client had initiated a stream abort, it will have a call
waiting for a reply in the queue. If more data continues to
arrive on the stream, the abort command could mistakenly get
signalled as complete. Remove the code from async data processing
that looked for waiting calls. Add a sanity check to ensure no
async call can ever be marked as needing a reply

* src/rpc/virnetclient.c: Ensure async data packets can't
  trigger a reply

Make it clearer that a log message is for an outgoing message

If a stream gets a server initiated abort, the client may still
send an abort request before it receives the server side abort.
This causes the server to send back another abort for the
stream. Since the protocol defines that abort is the last thing
to be sent, the client gets confused by this second abort from
the server. If the stream is already shutdown, just drop any
client requested abort, rather than sending back another message.
This fixes the regression from previous versions.

Tested as follows

In one virsh session

  virsh # start foo
  virsh # console foo

In other virsh session

  virsh # destroy foo

The first virsh session should be able to continue issuing
commands without error. Prior to this patch it saw

  virsh # list
  error: Failed to list active domains
  error: An error occurred, but the cause is unknown

  virsh # list
  error: Failed to list active domains
  error: no call waiting for reply with prog 536903814 vers 1 serial 9

* src/rpc/virnetserverprogram.c: Drop abort requests
  for streams which no longer exist