Since we are adding a new "per-hypervisor" protocol, we
make it so that the qemu remote protocol uses a new
PROTOCOL and PROGRAM number.  This allows us to easily
distinguish it from the normal REMOTE protocol.

This necessitates changing the proc in remote_message_header
from a "remote_procedure" to an "unsigned", which should
be the same size (and thus preserve the on-wire protocol).

Changes since v1:
 - Fixed up a couple of script problems in remote_generate_stubs.pl
 - Switch an int flag to a bool in dispatch.c

Changes since v2:
 - None

Changes since v3:
 - Change unsigned proc to signed proc, to conform to spec
Signed-off-by: NChris Lalancette <clalance@redhat.com>

There were some major, and some minor bugs having to do with
the reference counting of node devices in daemon/remote.c.

Some functions were completely failing to unreference node devices;
this would lead to many open file descriptors, which would eventually
fail.

The minor bugs were along the same lines, but were in rarely
used error paths.
Signed-off-by: NChris Lalancette <clalance@redhat.com>
Signed-off-by: NMatthias Bolte <matthias.bolte@googlemail.com>

Justin Clift reported a problem with adding virStoragePoolIsPersistent
to virsh's pool-info command, resulting in a strange problem. Here's
an example:

    virsh # pool-create-as images_dir3 dir - - - - "/home/images2"
    Pool images_dir3 created

    virsh # pool-info images_dir3
    Name:           images_dir3
    UUID:           90301885-94eb-4ca7-14c2-f30b25a29a36
    State:          running
    Capacity:       395.20 GB
    Allocation:     30.88 GB
    Available:      364.33 GB

    virsh # pool-destroy images_dir3
    Pool images_dir3 destroyed

At this point the images_dir3 pool should be gone (because it was
transient) and we should be able to create a new pool with the same name:

    virsh # pool-create-as images_dir3 dir - - - - "/home/images2"
    Pool images_dir3 created

    virsh # pool-info images_dir3
    Name:           images_dir3
    UUID:           90301885-94eb-4ca7-14c2-f30b25a29a36
    error: Storage pool not found

The new pool got the same UUID as the first one, but we didn't specify
one. libvirt should have picked a random UUID, but it didn't.

It turned out that virStoragePoolIsPersistent leaks a reference to the
storage pool object (actually remoteDispatchStoragePoolIsPersistent does).
As a result, pool-destroy doesn't remove the virStoragePool for the
"images_dir3" pool from the virConnectPtr's storagePools hash on libvirtd's
side. Then the second pool-create-as get's the stale virStoragePool object
associated with the "images_dir3" name. But this object has the old UUID.

This commit ensures that all get_nonnull_* and make_nonnull_* calls for
libvirt objects are matched properly with vir*Free calls. This fixes the
reference leaks and the reported problem.

All remoteDispatch*IsActive and remoteDispatch*IsPersistent functions were
affected. But also remoteDispatchDomainMigrateFinish2 was affected in the
success path. I wonder why that didn't surface earlier. Probably because
domainMigrateFinish2 is executed on the destination host and in the common
case this connection is opened especially for the migration and gets closed
after the migration is done. So there was no chance to run into a problem
because of the leaked reference.

Define the wire format for the new virDomainCreateWithFlags
API, and implement client and server side of marshaling code.

* daemon/remote.c (remoteDispatchDomainCreateWithFlags): Add
server side dispatch for virDomainCreateWithFlags.
* src/remote/remote_driver.c (remoteDomainCreateWithFlags)
(remote_driver): Client side serialization.
* src/remote/remote_protocol.x
(remote_domain_create_with_flags_args)
(remote_domain_create_with_flags_ret)
(REMOTE_PROC_DOMAIN_CREATE_WITH_FLAGS): Define wire format.
* daemon/remote_dispatch_args.h: Regenerate.
* daemon/remote_dispatch_prototypes.h: Likewise.
* daemon/remote_dispatch_table.h: Likewise.
* src/remote/remote_protocol.c: Likewise.
* src/remote/remote_protocol.h: Likewise.
* src/remote_protocol-structs: Likewise.

This introduces a new event type

   VIR_DOMAIN_EVENT_ID_IO_ERROR_REASON

This event is the same as the previous VIR_DOMAIN_ID_IO_ERROR
event, but also includes a string describing the cause of
the event.

Thus there is a new callback definition for this event type

typedef void (*virConnectDomainEventIOErrorReasonCallback)(virConnectPtr conn,
                                                           virDomainPtr dom,
                                                           const char *srcPath,
                                                           const char *devAlias,
                                                           int action,
                                                           const char *reason,
                                                           void *opaque);

This is currently wired up to the QEMU block IO error events

* daemon/remote.c: Dispatch IO error events to client
* examples/domain-events/events-c/event-test.c: Watch for
  IO error events
* include/libvirt/libvirt.h.in: Define new IO error event ID
  and callback signature
* src/conf/domain_event.c, src/conf/domain_event.h,
  src/libvirt_private.syms: Extend API to handle IO error events
* src/qemu/qemu_driver.c: Connect to the QEMU monitor event
  for block IO errors and emit a libvirt IO error event
* src/remote/remote_driver.c: Receive and dispatch IO error
  events to application
* src/remote/remote_protocol.x: Wire protocol definition for
  IO error events
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c: Watch for BLOCK_IO_ERROR event
  from QEMU monitor

* daemon/remote.c: Server side dispatcher
* daemon/remote_dispatch_args.h, daemon/remote_dispatch_prototypes.h,
  daemon/remote_dispatch_ret.h, daemon/remote_dispatch_table.h: Update
  with new API
* src/remote/remote_driver.c: Client side dispatcher
* src/remote/remote_protocol.c, src/remote/remote_protocol.h: Update
* src/remote/remote_protocol.x: Define new wire protocol

While running libvirtd under valgrind and doing some
snapshot testing I noticed that we would always leak a
connection reference.  The problem was actually that we
were leaking a domain reference in the libvirtd remote
snapshot code, which was in turn causing a leaked
connection reference.  Fix the situation by explicitly
taking and dropping a domain reference where we need it.
Signed-off-by: NChris Lalancette <clalance@redhat.com>

The user probably doesn't care what the gai error numbers are, as
much as what the failed conversion IP address was.

* src/remote/remote_driver.c (addrToString): Mention which address
could not be converted.
* daemon/remote.c (addrToString): Likewise.

Signed-off-by: NChris Lalancette <clalance@redhat.com>

* src/remote/remote_protocol.x src/remote/remote_protocol.h
  src/remote/remote_protocol.c src/remote/remote_driver.c: add the entry
  points in the remote driver
* daemon/remote.c daemon/remote_dispatch_args.h
  daemon/remote_dispatch_prototypes.h daemon/remote_dispatch_table.h:
  and implement the daemon counterpart

This patch adds the definition of the wire format for RPC calls
and implementation of the RPC client & server code
Signed-off-by: NStefan Berger <stefanb@us.ibm.com>

This defines the wire format for the new virDomainUpdateDeviceFlags()
API, and implements the server & client side of the marshalling code.

* daemon/remote.c: Server side dispatch for virDomainUpdateDeviceFlags
* src/remote/remote_driver.c: Client side serialization for
  virDomainUpdateDeviceFlags
* src/remote/remote_protocol.x: Define wire format for
  virDomainUpdateDeviceFlags
* daemon/remote_dispatch_args.h, daemon/remote_dispatch_prototypes.h,
  daemon/remote_dispatch_table.h, src/remote/remote_protocol.c,
  src/remote/remote_protocol.h: Re-generate code

This introduces a new event type

   VIR_DOMAIN_EVENT_ID_GRAPHICS

The same event can be emitted in 3 scenarios

  typedef enum {
      VIR_DOMAIN_EVENT_GRAPHICS_CONNECT = 0,
      VIR_DOMAIN_EVENT_GRAPHICS_INITIALIZE,
      VIR_DOMAIN_EVENT_GRAPHICS_DISCONNECT,
  } virDomainEventGraphicsPhase;

Connect/disconnect are triggered at socket accept/close.
The initialize phase is immediately after the protocol
setup and authentication has completed. ie when the
client is authorized and about to start interacting with
the graphical desktop

This event comes with *a lot* of potential information

 - IP address, port & address family of client
 - IP address, port & address family of server
 - Authentication scheme (arbitrary string)
 - Authenticated subject identity. A subject may have
   multiple identities with some authentication schemes.
   For example, vencrypt+sasl results in a x509dname
   and saslUsername identities.

This results in a very complicated callback :-(

   typedef enum {
      VIR_DOMAIN_EVENT_GRAPHICS_ADDRESS_IPV4,
      VIR_DOMAIN_EVENT_GRAPHICS_ADDRESS_IPV6,
   } virDomainEventGraphicsAddressType;

   struct _virDomainEventGraphicsAddress {
       int family;
       const char *node;
       const char *service;
   };
   typedef struct _virDomainEventGraphicsAddress virDomainEventGraphicsAddress;
   typedef virDomainEventGraphicsAddress *virDomainEventGraphicsAddressPtr;

   struct _virDomainEventGraphicsSubject {
      int nidentity;
      struct {
          const char *type;
          const char *name;
      } *identities;
   };
   typedef struct _virDomainEventGraphicsSubject virDomainEventGraphicsSubject;
   typedef virDomainEventGraphicsSubject *virDomainEventGraphicsSubjectPtr;

   typedef void (*virConnectDomainEventGraphicsCallback)(virConnectPtr conn,
                                                         virDomainPtr dom,
                                                         int phase,
                                                         virDomainEventGraphicsAddressPtr local,
                                                         virDomainEventGraphicsAddressPtr remote,
                                                         const char *authScheme,
                                                         virDomainEventGraphicsSubjectPtr subject,
                                                         void *opaque);

The wire protocol is similarly complex

   struct remote_domain_event_graphics_address {
     int family;
     remote_nonnull_string node;
     remote_nonnull_string service;
   };

   const REMOTE_DOMAIN_EVENT_GRAPHICS_IDENTITY_MAX = 20;

   struct remote_domain_event_graphics_identity {
     remote_nonnull_string type;
     remote_nonnull_string name;
   };

   struct remote_domain_event_graphics_msg {
     remote_nonnull_domain dom;
     int phase;
     remote_domain_event_graphics_address local;
     remote_domain_event_graphics_address remote;
     remote_nonnull_string authScheme;
     remote_domain_event_graphics_identity subject<REMOTE_DOMAIN_EVENT_GRAPHICS_IDENTITY_MAX>;
   };

This is currently implemented in QEMU for the VNC graphics
protocol, but designed to be usable with SPICE graphics in
the future too.

* daemon/remote.c: Dispatch graphics events to client
* examples/domain-events/events-c/event-test.c: Watch for
  graphics events
* include/libvirt/libvirt.h.in: Define new graphics event ID
  and callback signature
* src/conf/domain_event.c, src/conf/domain_event.h,
  src/libvirt_private.syms: Extend API to handle graphics events
* src/qemu/qemu_driver.c: Connect to the QEMU monitor event
  for VNC events and emit a libvirt graphics event
* src/remote/remote_driver.c: Receive and dispatch graphics
  events to application
* src/remote/remote_protocol.x: Wire protocol definition for
  graphics events
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c: Watch for VNC_CONNECTED,
  VNC_INITIALIZED & VNC_DISCONNETED events from QEMU monitor

This introduces a new event type

   VIR_DOMAIN_EVENT_ID_IO_ERROR

This event includes the action that is about to be taken
as a result of the watchdog triggering

  typedef enum {
     VIR_DOMAIN_EVENT_IO_ERROR_NONE = 0,
     VIR_DOMAIN_EVENT_IO_ERROR_PAUSE,
     VIR_DOMAIN_EVENT_IO_ERROR_REPORT,
  } virDomainEventIOErrorAction;

In addition it has the source path of the disk that had the
error and its unique device alias. It does not include the
target device name (/dev/sda), since this would preclude
triggering IO errors from other file backed devices (eg
serial ports connected to a file)

Thus there is a new callback definition for this event type

typedef void (*virConnectDomainEventIOErrorCallback)(virConnectPtr conn,
                                                     virDomainPtr dom,
                                                     const char *srcPath,
                                                     const char *devAlias,
                                                     int action,
                                                     void *opaque);

This is currently wired up to the QEMU block IO error events

* daemon/remote.c: Dispatch IO error events to client
* examples/domain-events/events-c/event-test.c: Watch for
  IO error events
* include/libvirt/libvirt.h.in: Define new IO error event ID
  and callback signature
* src/conf/domain_event.c, src/conf/domain_event.h,
  src/libvirt_private.syms: Extend API to handle IO error events
* src/qemu/qemu_driver.c: Connect to the QEMU monitor event
  for block IO errors and emit a libvirt IO error event
* src/remote/remote_driver.c: Receive and dispatch IO error
  events to application
* src/remote/remote_protocol.x: Wire protocol definition for
  IO error events
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c: Watch for BLOCK_IO_ERROR event
  from QEMU monitor

This introduces a new event type

   VIR_DOMAIN_EVENT_ID_WATCHDOG

This event includes the action that is about to be taken
as a result of the watchdog triggering

 typedef enum {
     VIR_DOMAIN_EVENT_WATCHDOG_NONE = 0,
     VIR_DOMAIN_EVENT_WATCHDOG_PAUSE,
     VIR_DOMAIN_EVENT_WATCHDOG_RESET,
     VIR_DOMAIN_EVENT_WATCHDOG_POWEROFF,
     VIR_DOMAIN_EVENT_WATCHDOG_SHUTDOWN,
     VIR_DOMAIN_EVENT_WATCHDOG_DEBUG,
 } virDomainEventWatchdogAction;

Thus there is a new callback definition for this event type

 typedef void (*virConnectDomainEventWatchdogCallback)(virConnectPtr conn,
                                                       virDomainPtr dom,
                                                       int action,
                                                       void *opaque);

* daemon/remote.c: Dispatch watchdog events to client
* examples/domain-events/events-c/event-test.c: Watch for
  watchdog events
* include/libvirt/libvirt.h.in: Define new watchdg event ID
  and callback signature
* src/conf/domain_event.c, src/conf/domain_event.h,
  src/libvirt_private.syms: Extend API to handle watchdog events
* src/qemu/qemu_driver.c: Connect to the QEMU monitor event
  for watchdogs and emit a libvirt watchdog event
* src/remote/remote_driver.c: Receive and dispatch watchdog
  events to application
* src/remote/remote_protocol.x: Wire protocol definition for
  watchdog events
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c: Watch for WATCHDOG event
  from QEMU monitor

This introduces a new event type

   VIR_DOMAIN_EVENT_ID_RTC_CHANGE

This event includes the new UTC offset measured in seconds.
Thus there is a new callback definition for this event type

 typedef void (*virConnectDomainEventRTCChangeCallback)(virConnectPtr conn,
                                                        virDomainPtr dom,
                                                        long long utcoffset,
                                                        void *opaque);

If the guest XML configuration for the <clock> is set to
offset='variable', then the XML will automatically be
updated with the new UTC offset value. This ensures that
during migration/save/restore the new offset is preserved.

* daemon/remote.c: Dispatch RTC change events to client
* examples/domain-events/events-c/event-test.c: Watch for
  RTC change events
* include/libvirt/libvirt.h.in: Define new RTC change event ID
  and callback signature
* src/conf/domain_event.c, src/conf/domain_event.h,
  src/libvirt_private.syms: Extend API to handle RTC change events
* src/qemu/qemu_driver.c: Connect to the QEMU monitor event
  for RTC changes and emit a libvirt RTC change event
* src/remote/remote_driver.c: Receive and dispatch RTC change
  events to application
* src/remote/remote_protocol.x: Wire protocol definition for
  RTC change events
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c: Watch for RTC_CHANGE event
  from QEMU monitor

The reboot event is not a normal lifecycle event, since the
virtual machine on the host does not change state. Rather the
guest OS is resetting the virtual CPUs. ie, the QEMU process
does not restart. Thus, this does not belong in the current
lifecycle events callback.

This introduces a new event type

    VIR_DOMAIN_EVENT_ID_REBOOT

It takes no parameters, besides the virDomainPtr, so it can
use the generic callback signature.

* daemon/remote.c: Dispatch reboot events to client
* examples/domain-events/events-c/event-test.c: Watch for
  reboot events
* include/libvirt/libvirt.h.in: Define new reboot event ID
* src/conf/domain_event.c, src/conf/domain_event.h,
  src/libvirt_private.syms: Extend API to handle reboot events
* src/qemu/qemu_driver.c: Connect to the QEMU monitor event
  for reboots and emit a libvirt reboot event
* src/remote/remote_driver.c: Receive and dispatch reboot
  events to application
* src/remote/remote_protocol.x: Wire protocol definition for
  reboot events

To avoid confusion, rename the current REMOTE_PROC_DOMAIN_EVENT
message to REMOTE_PROC_DOMAIN_EVENT_LIFECYCLE. This does not
cause ABI problems, since the names are only relevant at the source
code level. On the wire they encoding is a plain integer whose
value does not change

* src/remote/remote_protocol.x: Rename REMOTE_PROC_DOMAIN_EVENT
  to REMOTE_PROC_DOMAIN_EVENT_LIFECYCLE.
* daemon/remote.c, src/remote/remote_driver.c: Update code for
  renamed event

This wires up the remote driver to handle the new events APIs.
The public API allows an application to request a callback filters
events to a specific domain object, and register multiple callbacks
for the same event type. On the wire there are two strategies for
this

 - Register multiple callbacks with the remote daemon, each
   with filtering as needed
 - Register only one callback per event type, with no filtering

Both approaches have potential inefficiency. In the first scheme,
the same event gets sent over the wire many times if multiple
callbacks are registered. With the second scheme, unneccessary
events get sent over the wire if a per-domain filter is set on
the client. The second scheme is far easier to implement though,
so this patch takes that approach.

* daemon/dispatch.h: Don't export remoteRelayDomainEvent since it
  is no longer needed for unregistering callbacks, instead the
  unique callback ID is used
* daemon/libvirtd.c, daemon/libvirtd.h: Track and unregister
  callbacks based on callback ID, instead of function pointer
* daemon/remote.c: Switch over to using virConnectDomainEventRegisterAny
  instead of legacy virConnectDomainEventRegister function. Refactor
  remoteDispatchDomainEventSend() to cope with arbitrary event types
* src/driver.h, src/driver.c: Move verify() call into source file
  instead of header, to avoid polluting the global namespace with
  the verify function name
* src/remote/remote_driver.c: Implement new APIs for event
  registration. Refactor processCallDispatchMessage() to cope
  with arbitrary incoming event types. Merge remoteDomainQueueEvent()
  into processCallDispatchMessage() to avoid duplication of code.
  Rename remoteDomainReadEvent() to remoteDomainReadEventLifecycle()
* src/remote/remote_protocol.x: Define wire format for the new
  virConnectDomainEventRegisterAny and virConnectDomainEventDeregisterAny
  functions

* global: patch created by running:
for f in $(git ls-files '*.[ch]') ; do
    cppi $f > $f.t && mv $f.t $f
done

This defines the wire protocol for the new API

* src/remote/remote_protocol.x: Wire protocol definition
* src/remote/remote_driver.c,daemon/remote.c: Client and server
  side implementation
* daemon/remote_dispatch_args.h, daemon/remote_dispatch_prototypes.h,
  daemon/remote_dispatch_table.h, src/remote/remote_protocol.c,
  src/remote/remote_protocol.h: Re-generate from remote_protocol.x

* src/remote/remote_protocol.x: Define wire protocol format
  for virDomainGetJobInfo API
* src/remote/remote_driver.c, daemon/remote.c: Implement client
  and server marshalling code for virDomainGetJobInfo()
* daemon/remote_dispatch_args.h, daemon/remote_dispatch_prototypes.h
  daemon/remote_dispatch_ret.h, daemon/remote_dispatch_table.h,
  src/remote/remote_protocol.c, src/remote/remote_protocol.h: Rebuild
  files from src/remote/remote_protocol.x

It was used for error reporting only.

Server side dispatcher for Domain{Attach,Detach}DeviceFlags.

I noticed some debug messages are printed with an empty lines after
them. This patch removes these empty lines from all invocations of the
following macros:
    VIR_DEBUG
    VIR_DEBUG0
    VIR_ERROR
    VIR_ERROR0
    VIR_INFO
    VIR_WARN
    VIR_WARN0
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

The daemon will attempt to unregister domain events on client disconnect,
even if no events were ever registered. This raises an unneeded error.

Track in the qemu_client structure if events have been registered, and
check this when performing cleanup.

Many node device calls weren't properly relaying error messages, and
domain event registeration was not checking for error.

Use a dynamically sized xdr_array to pass memory stats on the wire.  This
supports the addition of future memory stats and reduces the message size
since only supported statistics are returned.

* src/remote/remote_protocol.x: provide defines for the new entry point
* src/remote/remote_driver.c daemon/remote.c: implement the client and
  server side
* daemon/remote_dispatch_args.h daemon/remote_dispatch_prototypes.h
  daemon/remote_dispatch_ret.h daemon/remote_dispatch_table.h
  src/remote/remote_protocol.c src/remote/remote_protocol.h: generated
  stubs

* src/remote/remote_protocol.x: update with new entry point
* daemon/remote.c: add the new server dispatcher
* daemon/remote_dispatch_args.h daemon/remote_dispatch_prototypes.h
  daemon/remote_dispatch_ret.h daemon/remote_dispatch_table.h
  src/remote/remote_protocol.c src/remote/remote_protocol.h: regenerated

There is currently no way to determine the libvirt version of a remote
libvirtd we are connected to. This is a useful piece of data to enable
feature detection.

This implements the virConnectIsSecure, virConnectIsEncrypted,
virDomainIsPersistent, virDomainIsActive, virNetworkIsActive,
virNetworkIsPersistent, virStoragePoolIsActive,
virStoragePoolIsPersistent, virInterfaceIsActive APIs in
(nearly) all drivers. Exceptions are:

 phyp: missing domainIsActive/Persistent
 esx: missing domainIsPersistent
 opennebula: missing domainIsActive/Persistent

* src/remote/remote_protocol.x: Define remote wire ABI for newly
  added APIs.
* daemon/remote_dispatch*.h: Re-generated from remote_protocol.x
* src/esx/esx_driver.c, src/lxc/lxc_driver.c, src/network/bridge_driver.c,
  src/opennebula/one_driver.c, src/openvz/openvz_conf.c,
  src/openvz/openvz_driver.c, src/phyp/phyp_driver.c,
  src/remote/remote_driver.c, src/storage/storage_driver.c,
  src/test/test_driver.c, src/uml/uml_driver.c, src/vbox/vbox_tmpl.c,
  src/xen/xen_driver.c, src/xen/xen_driver.h, src/xen/xen_inotify.c,
  src/xen/xen_inotify.h: Implement all the new APIs where possible

Since virMigratePrepareTunnel() is used for migration over the
native libvirt connection, there is never any need to pass the
target URI to this method.

* daemon/remote.c, src/driver.h, src/libvirt.c, src/libvirt_internal.h,
  src/qemu/qemu_driver.c, src/remote/remote_driver.c,
  src/remote/remote_protocol.c, src/remote/remote_protocol.h,
  src/remote/remote_protocol.x: Remove 'uri_in' parameter from
  virMigratePrepareTunnel() method

The code which updated the message length after writing the
payload wrote the updated length word in the wrong place since
the XDR object was given a buffer pointing to the start of the
header payload, rather than message start.

* daemon/remote.c: Fix updating of event message length so that
  we actually send the payload, not just the header

Implementation of tunnelled migration, using a Unix Domain Socket
on the qemu backend.  Note that this requires very new versions of
qemu (0.10.7 at least) in order to get the appropriate bugfixes.
Signed-off-by: NChris Lalancette <clalance@redhat.com>