Extend the RPC client code to allow file descriptors to be sent
to the server with calls, and received back with replies.

* src/remote/remote_driver.c: Stub extra args
* src/libvirt_private.syms, src/rpc/virnetclient.c,
  src/rpc/virnetclient.h, src/rpc/virnetclientprogram.c,
  src/rpc/virnetclientprogram.h: Extend APIs to allow
  FD passing

Define two new RPC message types VIR_NET_CALL_WITH_FDS and
VIR_NET_REPLY_WITH_FDS. These message types are equivalent
to VIR_NET_CALL and VIR_NET_REPLY, except that between the
message header, and payload there is a 32-bit integer field
specifying how many file descriptors have been passed.

The actual file descriptors are sent/recv'd out of band.

* src/rpc/virnetmessage.c, src/rpc/virnetmessage.h,
  src/libvirt_private.syms: Add support for handling
  passed file descriptors
* src/rpc/virnetprotocol.x: Extend protocol for FD
  passing

Add APIs to the virNetSocket object, to allow file descriptors
to be sent/received over UNIX domain socket connections

* src/rpc/virnetsocket.c, src/rpc/virnetsocket.h,
  src/libvirt_private.syms: Add APIs for FD send/recv

to escape the netcat command since it's passed to the shell. Adjust
expected test case output accordingly.

Based on a patch by Marc Deslauriers <marc.deslauriers@ubuntu.com>

RH: https://bugzilla.redhat.com/show_bug.cgi?id=562176
Ubuntu: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/517478
Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=573172

The libvirtd daemon had a few crude system tap probes. Some of
these were broken during the RPC rewrite. The new modular RPC
code is structured in a way that allows much more effective
tracing. Instead of trying to hook up the original probes,
define a new set of probes for the RPC and event code.

The master probes file is now src/probes.d.  This contains
probes for virNetServerClientPtr, virNetClientPtr, virSocketPtr
virNetTLSContextPtr and virNetTLSSessionPtr modules. Also add
probes for the poll event loop.

The src/dtrace2systemtap.pl script can convert the probes.d
file into a libvirt_probes.stp file to make use from systemtap
much simpler.

The src/rpc/gensystemtap.pl script can generate a set of
systemtap functions for translating RPC enum values into
printable strings. This works for all RPC header enums (program,
type, status, procedure) and also the authentication enum

The PROBE macro will automatically generate a VIR_DEBUG
statement, so any place with a PROBE can remove any existing
manual DEBUG statements.

* daemon/libvirtd.stp, daemon/probes.d: Remove obsolete probing
* daemon/libvirtd.h: Remove probe macros
* daemon/Makefile.am: Remove all probe buildings/install
* daemon/remote.c: Update authentication probes
* src/dtrace2systemtap.pl, src/rpc/gensystemtap.pl: Scripts
  to generate STP files
* src/internal.h: Add probe macros
* src/probes.d: Master list of probes
* src/rpc/virnetclient.c, src/rpc/virnetserverclient.c,
  src/rpc/virnetsocket.c, src/rpc/virnettlscontext.c,
  src/util/event_poll.c: Insert probe points, removing any
  DEBUG statements that duplicate the info

The virNetTLSContextRef API forgot to acquire/release the lock
while changing ctxt->refs

* src/rpc/virnettlscontext.c: Add lock calls

Pull the call to gnutls_x509_crt_get_dn up into a higher function
so that the 'dname' variable will be available for probe points

* src/rpc/virnettlscontext.c: Pull gnutls_x509_crt_get_dn up
  one level

Instead of directly manipulating sock->refs, add a virSocketRef
API

* src/rpc/virnetsocket.c, src/rpc/virnetsocket.h: Add virSocketRef

If we receive an error on the stream, set the EOF marker so
that any further (bogus) incoming data is dropped.

* src/rpc/virnetclientstream.c: Set EOF on stream

If we send back an unknown program error for async messages,
we will confuse the client because they only expect replies
for method calls. Just log & drop any invalid async messages

* src/rpc/virnetserver.c: Don't send error for async messages

Commit 597fe3ce accidentally
introduced a deadlock when reporting an unknown RPC program.
The virNetServerDispatchNewMessage method is called with
the client locked, and must therefore not attempt to send
any RPC messages back to the client. Only once the incoming
message is passed off to the virNetServerHandleJob worker
is it safe to start sending messages back

* src/rpc/virnetserver.c: Delay checking for unknown RPC
  program until in worker thread

Do not crash if virStreamFinish is called after error.

==11000== Invalid read of size 4
==11000==    at 0x373A8099A0: pthread_mutex_lock (pthread_mutex_lock.c:51)
==11000==    by 0x4C7CADE: virMutexLock (threads-pthread.c:85)
==11000==    by 0x4D57C31: virNetClientStreamRaiseError (virnetclientstream.c:203)
==11000==    by 0x4D385E4: remoteStreamFinish (remote_driver.c:3541)
==11000==    by 0x4D182F9: virStreamFinish (libvirt.c:14157)
==11000==    by 0x40FDC4: cmdScreenshot (virsh.c:3075)
==11000==    by 0x42BA40: vshCommandRun (virsh.c:14922)
==11000==    by 0x42ECCA: main (virsh.c:16381)
==11000==  Address 0x59b86c0 is 16 bytes inside a block of size 216 free'd
==11000==    at 0x4A06928: free (vg_replace_malloc.c:427)
==11000==    by 0x4C69E2B: virFree (memory.c:310)
==11000==    by 0x4D57B56: virNetClientStreamFree (virnetclientstream.c:184)
==11000==    by 0x4D3DB7A: remoteDomainScreenshot (remote_client_bodies.h:1812)
==11000==    by 0x4CFD245: virDomainScreenshot (libvirt.c:2903)
==11000==    by 0x40FB73: cmdScreenshot (virsh.c:3029)
==11000==    by 0x42BA40: vshCommandRun (virsh.c:14922)
==11000==    by 0x42ECCA: main (virsh.c:16381)

Mostly straight-forward, although this is the first API that
returns a new snapshot based on a snapshot rather than a domain.

* src/remote/remote_protocol.x
(REMOTE_PROC_DOMAIN_SNAPSHOT_GET_PARENT): New rpc.
(remote_domain_snapshot_get_parent_args)
(remote_domain_snapshot_get_parent_ret): New structs.
* src/rpc/gendispatch.pl: Adjust generator.
* src/remote/remote_driver.c (remote_driver): Use it.
* src/remote_protocol-structs: Update.

as they are not used with debugging turned off.

commit 984840a2 removed the
notification of waiting calls when VIR_NET_CONTINUE messages
arrive. This was to fix the case of a virStreamAbort() call
being prematurely notified of completion.

The problem is that sometimes there are dummy calls from a
virStreamRecv() call waiting that *do* need to be notified.

These dummy calls should have a status VIR_NET_CONTINUE. So
re-add the notification upon VIR_NET_CONTINUE, but only if
the waiter also has a status of VIR_NET_CONTINUE.

* src/rpc/virnetclient.c: Notify waiting call if stream data
  arrives
* src/rpc/virnetclientstream.c:  Mark dummy stream read packet
  with status VIR_NET_CONTINUE

Libvirt special-cases a specific VIR_ERR_RPC from the remote driver
back into VIR_ERR_NO_SUPPORT on the client, so that clients can
handle missing rpc functions the same whether the hypervisor driver
is local or remote.  However, commit c1b22644 introduced a regression:
VIR_FROM_THIS changed from VIR_FROM_REMOTE to VIR_FROM_RPC, so the
special casing no longer works if the server uses the newer error
domain.

* src/rpc/virnetclientprogram.c
(virNetClientProgramDispatchError): Also cater to 0.9.3 and newer.

So we can allow GSSAPI authentication for ssh.
Signed-off-by: NMatthias Witte <witte@netzquadrat.de>

* src/rpc/virnettlscontext.c: fix memory leak on
  virNetTLSContextValidCertificate.

* Detected in valgrind run:

==25667==
==25667== 6,085 (44 direct, 6,041 indirect) bytes in 1 blocks are definitely
lost in loss record 326 of 351
==25667==    at 0x4005447: calloc (vg_replace_malloc.c:467)
==25667==    by 0x4F2791F3: _asn1_add_node_only (structure.c:53)
==25667==    by 0x4F27997A: _asn1_copy_structure3 (structure.c:421)
==25667==    by 0x4F276A50: _asn1_append_sequence_set (element.c:144)
==25667==    by 0x4F2743FF: asn1_der_decoding (decoding.c:1194)
==25667==    by 0x4F22B9CC: gnutls_x509_crt_import (x509.c:229)
==25667==    by 0x805274B: virNetTLSContextCheckCertificate
(virnettlscontext.c:1009)
==25667==    by 0x804DE32: testTLSSessionInit (virnettlscontexttest.c:693)
==25667==    by 0x804F14D: virtTestRun (testutils.c:140)
==25667==
==25667== 23,188 (88 direct, 23,100 indirect) bytes in 11 blocks are definitely
lost in loss record 346 of 351
==25667==    at 0x4005447: calloc (vg_replace_malloc.c:467)
==25667==    by 0x4F22B841: gnutls_x509_crt_init (x509.c:50)
==25667==    by 0x805272B: virNetTLSContextCheckCertificate
(virnettlscontext.c:1003)
==25667==    by 0x804DDD1: testTLSSessionInit (virnettlscontexttest.c:673)
==25667==    by 0x804F14D: virtTestRun (testutils.c:140)

* How to reproduce?
% cd libvirt && ./configure && make && make -C tests valgrind
or
% valgrind -v --leak-check=full ./tests/virnettlscontexttest
Signed-off-by: NAlex Jia <ajia@redhat.com>

This patch annotates APIs with low or high priority.
In low set MUST be all APIs which might eventually access monitor
(and thus block indefinitely). Other APIs may be marked as high
priority. However, some must be (e.g. domainDestroy).

For high priority calls (HPC), there are some high priority workers
(HPW) created in the pool. HPW can execute only HPC, although normal
worker can process any call regardless priority. Therefore, only those
APIs which are guaranteed to end in reasonable small amount of time
can be marked as HPC.

The size of this HPC pool is static, because HPC are expected to end
quickly, therefore jobs assigned to this pool will be served quickly.
It can be configured in libvirtd.conf via prio_workers variable.
Default is set to 5.

To mark API with low or high priority, append priority:{low|high} to
it's comment in src/remote/remote_protocol.x. This is similar to
autogen|skipgen. If not marked, the generator assumes low as default.

Includes impl of python binding since the generator was not
able to cope.

Note: Requires gendispatch.pl patch from Matthias Bolte

https://www.redhat.com/archives/libvir-list/2011-August/msg01367.html

Commit 2c85644b attempted to
fix a problem with tracking RPC messages from streams by doing

-            if (msg->header.type == VIR_NET_REPLY) {
+            if (msg->header.type == VIR_NET_REPLY ||
+                (msg->header.type == VIR_NET_STREAM &&
+                 msg->header.status != VIR_NET_CONTINUE)) {
                 client->nrequests--;

In other words any stream packet, with status NET_OK or NET_ERROR
would cause nrequests to be decremented. This is great if the
packet from from a synchronous virStreamFinish or virStreamAbort
API call, but wildly wrong if from a server initiated abort.
The latter resulted in 'nrequests' being decremented below zero.
This then causes all I/O for that client to be stopped.

Instead of trying to infer whether we need to decrement the
nrequests field, from the message type/status, introduce an
explicit 'bool tracked' field to mark whether the virNetMessagePtr
object is subject to tracking.

Also add a virNetMessageClear function to allow a message
contents to be cleared out, without adversely impacting the
'tracked' field as a naive memset() would do

* src/rpc/virnetmessage.c, src/rpc/virnetmessage.h: Add
  a 'bool tracked' field and virNetMessageClear() API
* daemon/remote.c, daemon/stream.c, src/rpc/virnetclientprogram.c,
  src/rpc/virnetclientstream.c, src/rpc/virnetserverclient.c,
  src/rpc/virnetserverprogram.c: Switch over to use
  virNetMessageClear() and pass in the 'bool tracked' value
  when creating messages.

The bufferOffset has been initialized to zero in virNetMessageEncodePayloadRaw(),
so, we use bufferLength to represent the length of message which is going to be
sent to client side.

From: Matthias Bolte <matthias.bolte@googlemail.com>
Tested-by: NJim Fehlig <jfehlig@novell.com>

Matthias provided this patch to fix an issue I encountered in the
generator with APIs containing call-by-ref long type, e.g.

int virDomainMigrateGetMaxSpeed(virDomainPtr domain,
                                unsigned long *bandwidth,
                                unsigned int flags);

In case we add a new program in the future (we did that in the past and
we are going to do it again soon) current daemon will behave badly with
new client that wants to use the new program. Before the RPC rewrite we
used to just send an error reply to any request with unknown program.
With the RPC rewrite in 0.9.3 the daemon just closes the connection
through which such request was sent. This patch fixes this regression.

My previous patch 74c75671
introduced a regression by removing TLS initialization from client.

When spice_tls is set but listen_tls is not, we don't initialize
GnuTLS library. So any later gnutls call (e.g. during migration,
where we initialize a certificate) will access uninitialized GnuTLS
internal structs and throws an error.

Although, we might now initialize GnuTLS twice, it is safe according
to the documentation:

    This function can be called many times,
    but will only do something the first time.

This patch creates 2 functions: virNetTLSInit and virNetTLSDeinit
with respect to written above.

Regression introduced in commit 5d30db09.

* src/rpc/virnetsocket.c (virNetSocketNewListenUNIX) [WIN32]: Use
correct signature.

If a client had initiated a stream abort, it will have a call
waiting for a reply in the queue. If more data continues to
arrive on the stream, the abort command could mistakenly get
signalled as complete. Remove the code from async data processing
that looked for waiting calls. Add a sanity check to ensure no
async call can ever be marked as needing a reply

* src/rpc/virnetclient.c: Ensure async data packets can't
  trigger a reply

Make it clearer that a log message is for an outgoing message

If a stream gets a server initiated abort, the client may still
send an abort request before it receives the server side abort.
This causes the server to send back another abort for the
stream. Since the protocol defines that abort is the last thing
to be sent, the client gets confused by this second abort from
the server. If the stream is already shutdown, just drop any
client requested abort, rather than sending back another message.
This fixes the regression from previous versions.

Tested as follows

In one virsh session

  virsh # start foo
  virsh # console foo

In other virsh session

  virsh # destroy foo

The first virsh session should be able to continue issuing
commands without error. Prior to this patch it saw

  virsh # list
  error: Failed to list active domains
  error: An error occurred, but the cause is unknown

  virsh # list
  error: Failed to list active domains
  error: no call waiting for reply with prog 536903814 vers 1 serial 9

* src/rpc/virnetserverprogram.c: Drop abort requests
  for streams which no longer exist

Every active stream results in a reference being held on the
virNetServerClientPtr object. This meant that if a client quit
with any streams active, although all I/O was stopped the
virNetServerClientPtr object would leak. This causes libvirtd
to leak any file handles associated with open streams when a
client quit

To fix this, when we call virNetServerClientClose there is a
callback invoked which lets the daemon release the streams
and thus the extra references

* daemon/remote.c: Add a hook to close all streams
* daemon/stream.c, daemon/stream.h: Add API for releasing
  all streams
* src/rpc/virnetserverclient.c, src/rpc/virnetserverclient.h:
  Allow registration of a hook to trigger when closing client

This patch allows owner's UID to be changed as well.

So that callers can change the default value.

* src/rpc/virnetsocket.c, src/rpc/virnetsocket.h: Add
  virNetSocketDupFD()
* src/rpc/virnetclient.c, src/rpc/virnetclient.h: Add
  virNetClientDupFD() and virNetClientGetFD()

When trying to use any SASL authentication for TCP sockets by
setting auth_tls = "sasl" in libvirtd.conf on server side, the
client will hang because of the sasl session relocking other than
dropping the lock when exiting virNetSASLSessionExtKeySize()
* src/rpc/virnetsaslcontext.c: virNetSASLSessionExtKeySize drop the
  lock on exit

This patch introduces a internal RPC API "virNetServerClose", which
is standalone with "virNetServerFree".  it closes all the socket fds,
and unlinks the unix socket paths, regardless of whether the socket
is still referenced or not.

This is to address regression bug:
https://bugzilla.redhat.com/show_bug.cgi?id=725702

In virNetServerNew, Coverity didn't realize that srv->mdsnGroupName
can only be non-NULL if mdsnGroupName was non-NULL.

In virNetServerRun, Coverity didn't realize that the array is non-NULL
if the array count is non-zero.

* src/rpc/virnetserver.c (virNetServerNew): Use alternate pointer.
(virNetServerRun): Give coverity a hint.

Detected by Coverity.

* src/rpc/virnetserverclient.c (virNetServerClientDispatchRead):
Avoid null deref on OOM.

Detected by Coverity.  Freeing the wrong variable results in both
a memory leak and the likelihood of the caller dereferencing through
a freed pointer.

* src/rpc/virnettlscontext.c (virNetTLSSessionNew): Free correct
variable.