New optional parameter "keyfile" for ssh transport allows the user to select
the private key to be used to authenticate to the remote host.

There is some commonality between the code for sanity checking
certs when initializing libvirt and the code for validating
certs during a live TLS session handshake. This patchset splits
up the sanity checking function into several smaller functions
each doing a specific type of check. The cert validation code
is then updated to also call into these functions

* src/rpc/virnettlscontext.c: Refactor cert validation code

The gnutls_certificate_type_set_priority method is deprecated.
Since we already set the default gnutls priority, it was not
serving any useful purpose and can be removed

* src/rpc/virnettlscontext.c: Remove gnutls_certificate_type_set_priority
  call

If the virStateInitialize call fails we must shutdown libvirtd
since drivers will not be available. Just free'ing the virNetServer
is not sufficient, we must send a SIGTERM to ourselves so that
we interrupt the event loop and trigger a orderly shutdown

* daemon/libvirtd.c: Kill ourselves if state init fails
* src/rpc/virnetserver.c: Add some debugging to event loop

The generator can handle everything except virDomainGetBlockJobInfo().

* src/remote/remote_protocol.x: provide defines for the new entry points
* src/remote/remote_driver.c daemon/remote.c: implement the client and
  server side for virDomainGetBlockJobInfo.
* src/remote_protocol-structs: structure definitions for protocol verification
* src/rpc/gendispatch.pl: Permit some unsigned long parameters

In preparation for a future patch adding new virFile APIs.

* src/util/files.h, src/util/files.c: Move...
* src/util/virfile.h, src/util/virfile.c: ...here, and rename
functions to virFile prefix.  Macro names are intentionally
left alone.
* *.c: All '#include "files.h"' uses changed.
* src/Makefile.am (UTIL_SOURCES): Reflect rename.
* cfg.mk (exclude_file_name_regexp--sc_prohibit_close): Likewise.
* src/libvirt_private.syms: Likewise.
* docs/hacking.html.in: Likewise.
* HACKING: Regenerate.

The only 'void name(void)' style procedure in the protocol is 'close' that
is handled special, but also programming errors like a missing _args or
_ret suffix on the structs in the .x files can create such a situation by
accident. Making the generator aware of this avoids bogus errors from the
generator such as:

  Use of uninitialized value in exists at ./rpc/gendispatch.pl line 967.

Also this allows to get rid of the -c option and the special case code for
the 'close' procedure, as the generator handles it now correctly.

Reported by Michal Privoznik

Though we prefer users to have SSH keys setup, virt-manager users still
depend on remote SSH connections to launch a password dialog. This fixes
launch ssh-askpass

Fix suggested by danpb

* src/rpc/virnetclient.c: Skip close attempt if virNetClientPtr
  is NULL

If a key purpose or usage field is marked as non-critical in the
certificate, then a data mismatch is not (ordinarily) a cause for
rejecting the connection

* src/rpc/virnettlscontext.c: Honour key usage/purpose criticality

If key usage or purpose data is not present in the cert, the
RFC recommends that access be allowed. Also fix checking of
key usage to include requirements for client/server certs,
and fix key purpose checking to treat data as a list of bits

* src/rpc/virnettlscontext.c: Fix mixed up error messages

Gnutls requires that certificates have basic constraints present
to be used as a CA certificate. OpenSSL doesn't add this data
by default, so add a sanity check to catch this situation. Also
validate that the key usage and key purpose constraints contain
correct data

* src/rpc/virnettlscontext.c: Add sanity checking of certificate
  constraints

If the libvirt daemon or libvirt client is configured with bogus
certificates, it is very unhelpful to only find out about this
when a TLS connection is actually attempted. Not least because
the error messages you get back for failures are incredibly
obscure.

This adds some basic sanity checking of certificates at the
time the virNetTLSContext object is created. This is at libvirt
startup, or when creating a virNetClient instance.

This checks that the certificate expiry/start dates are valid
and that the certificate is actually signed by the CA that is
loaded.

* src/rpc/virnettlscontext.c: Add certificate sanity checks

Since the I/O callback registered against virNetSocket will
hold a reference on the virNetClient, we can't rely on the
virNetClientFree to be able to close the network connection.
The last reference will only go away when the event callback
fires (likely due to EOF from the server).

This is sub-optimal and can potentially cause a leak of the
virNetClient object if the server were to not explicitly
close the socket itself

* src/remote/remote_driver.c: Explicitly close the client
  object when disconnecting
* src/rpc/virnetclient.c, src/rpc/virnetclient.h: Add a
  virNetClientClose method

When unregistering an I/O callback from a virNetSocket object,
there is still a chance that an event may come in on the callback.
In this case it is possible that the virNetSocket might have been
freed already. Make use of a virFreeCallback when registering
the I/O callbacks and hold a reference for the entire time the
callback is set.

* src/rpc/virnetsocket.c: Register a free function for the
  file handle watch
* src/rpc/virnetsocket.h, src/rpc/virnetserverservice.c,
  src/rpc/virnetserverclient.c, src/rpc/virnetclient.c: Add
  a free function for the socket I/O watches

Remove the need for a virNetSocket object to be protected by
locks from the object using it, by introducing its own native
locking and reference counting

* src/rpc/virnetsocket.c: Add locking & reference counting

* src/rpc/virnetclient.c: Add debugging of ref counts

If we get an I/O error in the async event callback for an RPC
client, we might not have consumed all pending data off the
wire. This could result in the callback being immediately
invoked again. At which point the same I/O might occur. And
we're invoked again. And again...And again...

Unregistering the async event callback if an error occurs is
a good safety net. The real error will be seen when the next
RPC method is invoked

* src/rpc/virnetclient.c: Unregister event callback on error

These typos are introduced by file renaming in commit b17b4afa.

src/remote/qemu_protocol.x \
src/remote/remote_protocol.x \
src/rpc/gendispatch.pl:
    s/remote_generator/gendispatch/

src/rpc/genprotocol.pl:
    s/remote\/remote_protocol/remote_protocol/

* src/rpc/virnettlscontext.c: s/read/write/

If the server succesfully validates the client cert, it will send
back a single byte, under TLS. If it fails, it will close the
connection. In this case, we were just reporting the standard
I/O error. The original RPC code had a special case hack for the
GNUTLS_E_UNEXPECTED_PACKET_LENGTH error code to make us report
a more useful error message

* src/rpc/virnetclient.c: Return ENOMSG if we get
  GNUTLS_E_UNEXPECTED_PACKET_LENGTH
* src/rpc/virnettlscontext.c: Report cert failure if we
  see ENOMSG

Rather than trying to clean up the ssh child ourselves, and risk
subtle differences from the socket creation error path, we can
just use the new APIs.

* src/rpc/virnetsocket.c (virNetSocketFree): Use new function.

Continuation of commit 313ac7fd, and enforce things with a syntax
check.

Technically, virNetServerClientCalculateHandleMode is not printing
a mode_t, but rather a collection of VIR_EVENT_HANDLE_* bits;
however, these bits are < 8, so there is no different in the
output, and that was the easiest way to silence the new syntax check.

* cfg.mk (sc_flags_debug): New syntax check.
(exclude_file_name_regexp--sc_flags_debug): Add exemptions.
* src/fdstream.c (virFDStreamOpenFileInternal): Print flags in
hex, mode_t in octal.
* src/libvirt-qemu.c (virDomainQemuMonitorCommand)
(virDomainQemuAttach): Likewise.
* src/locking/lock_driver_nop.c (virLockManagerNopInit): Likewise.
* src/locking/lock_driver_sanlock.c (virLockManagerSanlockInit):
Likewise.
* src/locking/lock_manager.c: Likewise.
* src/qemu/qemu_migration.c: Likewise.
* src/qemu/qemu_monitor.c: Likewise.
* src/rpc/virnetserverclient.c
(virNetServerClientCalculateHandleMode): Print mode with %o.

When replacing the default SEGV/ABORT/BUS signal handlers you
can't rely on the process being terminated after your custom
handler runs. It is neccessary to manually restore the default
handler and then re-raise the signal

* src/rpc/virnetserver.c: Restore default handler and raise
  signal

* src/rpc/virnetserverprogram.h: Remove unused typedef for
  virNetServerProgramErrorHander function callback
* daemon/remote.h: Remove decl for non-existant variables

connectDBus is only used if HAVE_DBUS is set. Therefore mark
it as potentially unused.

V2: Remove policy kit references from virNetServer and use DBus APIs
    directly, if available.

This tweaks the RPC generator to cope with some naming
conventions used for the QEMU specific APIs

* daemon/remote.c: Server side dispatcher
* src/remote/remote_driver.c: Client side dispatcher
* src/remote/qemu_protocol.x: Wire protocol definition
* src/rpc/gendispatch.pl: Use '$structprefix' in method
  names, fix QEMU flags and fix dispatcher method names

Set StrictHostKeyChecking=no to auto-accept new ssh host keys if the
no_verify extra parameter was specified.  This won't disable host key
checking for already known hosts.  Includes a test and documentation.

Add missing line break and fix indention level.

Reported by Cole Robinson.

The dispatch for the CLOSE RPC call was invoking the method
virNetServerClientClose(). This caused the client connection
to be immediately terminated. This meant the reply to the
final RPC message was never sent. Prior to the RPC rewrite
we merely flagged the connection for closing, and actually
closed it when the next RPC call dispatch had completed.

* daemon/remote.c: Flag connection for a delayed close
* daemon/stream.c: Update to use new API for closing
  failed connection
* src/rpc/virnetserverclient.c, src/rpc/virnetserverclient.h:
  Add support for a delayed connection close. Rename the
  virNetServerClientMarkClose method to virNetServerClientImmediateClose
  to clarify its semantics

When sending back the final OK or ERROR message on completion
of a stream, we were not decrementing the 'nrequests' tracker
on the client. With the default requests limit of '5', this
meant once a client had created 5 streams, they are unable to
process any further RPC calls.  There was also a bug when
handling an error from decoding a message length header, which
meant a client connection would not immediately be closed.

* src/rpc/virnetserverclient.c: Fix release of request after
  stream completion & mark client for close on error

In one exit path we forgot to free the virNetMessage object causing
a large memory leak for streams which send a lot of data. Some other
paths were calling VIR_FREE directly instead of virNetMessageFree
although this was (currently) harmless.

* src/rpc/virnetclientstream.c: Fix leak of msg object
* src/rpc/virnetclientprogram.c: Call virNetMessageFree instead
  of VIR_FREE

The virNetTLSContextNew was being passed key/cert parameters in
the wrong order. This wasn't immediately visible because if
virNetTLSContextNewPath was used, a second bug reversed the order
of those parameters again.

Only if the paths were manually specified in /etc/libvirt/libvirtd.conf
did the bug appear

* src/rpc/virnettlscontext.c: Fix order of params passed to
  virNetTLSContextNew

Gnulib finally learned how to do pthread_sigmask on mingw.

* .gnulib: Update to latest, for pthread_sigmask.
* bootstrap.conf (gnulib_modules): Add pthread_sigmask.
* configure.ac (AC_CHECK_FUNCS): Drop redundant check.
* src/rpc/virnetclient.c (virNetClientSetTLSSession)
(virNetClientIOEventLoop): Make code unconditional.
* src/util/command.c (virFork): Likewise.
* tools/virsh.c (doMigrate, cmdMigrate): Likewise.

Coverity noted that 4 out of 5 calls to virNetClientStreamRaiseError
checked the return value.  This case expects a particular value, so
warn if our expectations went wrong due to some bug elsewhere.

* src/rpc/virnetclient.c (virNetClientCallDispatchStream): Warn on
  unexpected scenario.

Detected by Coverity.  The leak is on an error path, but I'm not
sure whether that path is likely to be triggered in practice.

* src/rpc/virnetserverservice.c (virNetServerServiceAccept): Plug leak.

Spotted by Coverity.  If we don't update tmp each time through
the loop, then if the filter being removed was not the head of
the list, we accidentally lose all filters prior to the one we
wanted to remove.

* src/rpc/virnetserverclient.c (virNetServerClientRemoveFilter):
    Don't lose unrelated filters.