Followup to https://bugzilla.redhat.com/show_bug.cgi?id=599091,
commit 20206a4b, to reduce disk waste in padding.

* src/qemu/qemu_monitor.h (QEMU_MONITOR_MIGRATE_TO_FILE_BS): Drop
back to 4k.
(QEMU_MONITOR_MIGRATE_TO_FILE_TRANSFER_SIZE): New macro.
* src/qemu/qemu_driver.c (qemudDomainSaveFlag): Update comment.
* src/qemu/qemu_monitor_text.c (qemuMonitorTextMigrateToFile): Use
two invocations of dd to output non-aligned large blocks.
* src/qemu/qemu_monitor_json.c (qemuMonitorJSONMigrateToFile):
Likewise.

This patch adds an optional XML attribute to a nwfilter rule to give the user control over whether the rule is supposed to be using the iptables state match or not. A rule may now look like shown in the XML below with the statematch attribute either having value '0' or 'false' (case-insensitive).

[...]
<rule action='accept' direction='in' statematch='false'>
<tcp srcmacaddr='1:2:3:4:5:6'
           srcipaddr='10.1.2.3' srcipmask='32'
           dscp='33'
           srcportstart='20' srcportend='21'
           dstportstart='100' dstportend='1111'/>
</rule>
[...]

I am also extending the nwfilter schema and add this attribute to a test case.

This patch adds the persistence status (yes/no) to the output of the virsh
dominfo and pool-info commands.  This patch also adds the autostart status
to the output of the virsh pool-info command.

Red Hat BZ for this:

  https://bugzilla.redhat.com/show_bug.cgi?id=603696

Use virBuffer* API to conditionally keep the portion of the command
line specific to HMC, so that IVM can work.
Signed-off-by: NEric Blake <eblake@redhat.com>

The following patch enables the iptables match target to be used by
default for incoming traffic. So far it has only be used for outgoing
traffic.

This patch works around a recent extension of the netlink driver I had made use of when building the netlink messages. Unfortunately older kernels don't accept IFLA_IFNAME + name of interface as a replacement for the interface's index, so this patch now gets the interface index ifindex if it's not provided (ifindex <= 0).

Trivial fix changing printf() calls to vshPrint() where the ctl
variable is available.

Improves the help text for vol-path, vol-name, and vol-key, which
previously referred to volume UUIDs.

Addresses BZ # 598365.

Presently the vol-key command only supports being provided with
a volume path.

This patch adds support for providing it with a pool and volume
identifier pair as well.

    virsh # vol-key --pool <pool-name-or-uuid> <vol-name-or-path>

Justin Clift reported a problem with adding virStoragePoolIsPersistent
to virsh's pool-info command, resulting in a strange problem. Here's
an example:

    virsh # pool-create-as images_dir3 dir - - - - "/home/images2"
    Pool images_dir3 created

    virsh # pool-info images_dir3
    Name:           images_dir3
    UUID:           90301885-94eb-4ca7-14c2-f30b25a29a36
    State:          running
    Capacity:       395.20 GB
    Allocation:     30.88 GB
    Available:      364.33 GB

    virsh # pool-destroy images_dir3
    Pool images_dir3 destroyed

At this point the images_dir3 pool should be gone (because it was
transient) and we should be able to create a new pool with the same name:

    virsh # pool-create-as images_dir3 dir - - - - "/home/images2"
    Pool images_dir3 created

    virsh # pool-info images_dir3
    Name:           images_dir3
    UUID:           90301885-94eb-4ca7-14c2-f30b25a29a36
    error: Storage pool not found

The new pool got the same UUID as the first one, but we didn't specify
one. libvirt should have picked a random UUID, but it didn't.

It turned out that virStoragePoolIsPersistent leaks a reference to the
storage pool object (actually remoteDispatchStoragePoolIsPersistent does).
As a result, pool-destroy doesn't remove the virStoragePool for the
"images_dir3" pool from the virConnectPtr's storagePools hash on libvirtd's
side. Then the second pool-create-as get's the stale virStoragePool object
associated with the "images_dir3" name. But this object has the old UUID.

This commit ensures that all get_nonnull_* and make_nonnull_* calls for
libvirt objects are matched properly with vir*Free calls. This fixes the
reference leaks and the reported problem.

All remoteDispatch*IsActive and remoteDispatch*IsPersistent functions were
affected. But also remoteDispatchDomainMigrateFinish2 was affected in the
success path. I wonder why that didn't surface earlier. Probably because
domainMigrateFinish2 is executed on the destination host and in the common
case this connection is opened especially for the migration and gets closed
after the migration is done. So there was no chance to run into a problem
because of the leaked reference.

Commit b9efc7dc made virFileHasSuffix
case insensitive. Honor this in the tests by switching vmdk to VMDK.

* src/phyp/phyp_driver.c (phypNumDomainsGeneric): Avoid glob
collision by quoting sed argument.
(phypDomainSetCPU): Avoid non-portable \+ in sed.
(phypGetVIOSPartitionID, phypDiskType, phypListDomainsGeneric)
(phypListDefinedDomains): Micro-optimize anchored substitutions.

This is a trivial fix for several autostart yes/no strings that
weren't correctly marked for translation.

Make 'start --paused' mirror 'create --paused'.

* tools/virsh.c (cmdStart): Use new virDomainCreateWithFlags API
when needed.
* tools/virsh.pod (start): Document --paused.

Match earlier change for qemu pause support with virDomainCreateXML.

* src/qemu/qemu_driver.c (qemudDomainObjStart): Add parameter; all
callers changed.
(qemudDomainStartWithFlags): Implement flag support.

* src/esx/esx_driver.c (esxDomainCreate): Move guts...
(esxDomainCreateWithFlags): ...to new function.
(esxDriver): Trivially support the new API.
* src/lxc/lxc_driver.c (lxcDomainStart, lxcDomainStartWithFlags)
(lxcDriver): Likewise.
* src/opennebula/one_driver.c (oneDomainStart)
(oneDomainStartWithFlags, oneDriver): Likewise.
* src/openvz/openvz_driver.c (openvzDomainCreate)
(openvzDomainCreateWithFlags, openvzDriver): Likewise.
* src/qemu/qemu_driver.c (qemudDomainStart)
(qemudDomainStartWithFlags, qemuDriver): Likewise.
* src/test/test_driver.c (testDomainCreate)
(testDomainCreateWithFlags, testDriver): Likewise.
* src/uml/uml_driver.c (umlDomainStart, umlDomainStartWithFlags)
(umlDriver): Likewise.
* src/vbox/vbox_tmpl.c (vboxDomainCreate)
(vboxDomainCreateWithFlags, Driver): Likewise.
* src/xen/xen_driver.c (xenUnifiedDomainCreate)
(xenUnifiedDomainCreateWithFlags, xenUnifiedDriver): Likewise.
* src/xenapi/xenapi_driver.c (xenapiDomainCreate)
(xenapiDomainCreateWithFlags, xenapiDriver): Likewise.

Define the wire format for the new virDomainCreateWithFlags
API, and implement client and server side of marshaling code.

* daemon/remote.c (remoteDispatchDomainCreateWithFlags): Add
server side dispatch for virDomainCreateWithFlags.
* src/remote/remote_driver.c (remoteDomainCreateWithFlags)
(remote_driver): Client side serialization.
* src/remote/remote_protocol.x
(remote_domain_create_with_flags_args)
(remote_domain_create_with_flags_ret)
(REMOTE_PROC_DOMAIN_CREATE_WITH_FLAGS): Define wire format.
* daemon/remote_dispatch_args.h: Regenerate.
* daemon/remote_dispatch_prototypes.h: Likewise.
* daemon/remote_dispatch_table.h: Likewise.
* src/remote/remote_protocol.c: Likewise.
* src/remote/remote_protocol.h: Likewise.
* src/remote_protocol-structs: Likewise.

Persistent domain creation needs the same features as transient
domains, but virDomainCreate lacks the flags argument present in
virDomainCreateXML.  virDomainCreateFlags is already claimed as
a public enum, so we have to break convention and expose
virDomainCreateWithFlags.

* include/libvirt/libvirt.h.in (virDomainCreateWithFlags): Add.
* src/driver.h (virDrvDomainCreateWithFlags): Internal API.
* src/libvirt.c (virDomainCreateWithFlags): Glue public API to
driver API.
* src/libvirt_public.syms (LIBVIRT_0.8.2): Expose public API.
* src/esx/esx_driver.c (esxDriver): Add stub for driver.
* src/lxc/lxc_driver.c (lxcDriver): Likewise.
* src/opennebula/one_driver.c (oneDriver): Likewise.
* src/openvz/openvz_driver.c (openvzDriver): Likewise.
* src/phyp/phyp_driver.c (phypDriver): Likewise.
* src/qemu/qemu_driver.c (qemuDriver): Likewise.
* src/remote/remote_driver.c (remote_driver): Likewise.
* src/test/test_driver.c (testDriver): Likewise.
* src/uml/uml_driver.c (umlDriver): Likewise.
* src/vbox/vbox_tmpl.c (Driver): Likewise.
* src/xen/xen_driver.c (xenUnifiedDriver): Likewise.
* src/xenapi/xenapi_driver.c (xenapiDriver): Likewise.

* .hgignore: Delete, no longer used.
* examples/python/.gitignore: Delete, covered globally.
* include/.gitignore: Likewise.
* python/tests/.gitignore: Likewise.
* docs/schemas/.gitignore: Likewise.
* tests/xml2sexprdata/.gitignore: Likewise.
* tests/sexpr2xmldata/.gitignore: Likewise.
* tests/confdata/.gitignore: Likewise.
* tests/xencapsdata/.gitignore: Likewise.
* tests/xmconfigdata/.gitignore: Likewise.
* tests/xml2sexprdata/.gitignore: Likewise.

Daniel's patch works with gcc and CFLAGS containing -O (the
autoconf default), but fails with non-gcc or with other
CFLAGS (such as -g), since c-ctype.h declares c_isdigit as
a macro only for certain compilation settings.

* src/Makefile.am (libvirt_parthelper_LDFLAGS): Add gnulib
library, for when c_isdigit is not a macro.
* src/storage/parthelper.c (main): Avoid out-of-bounds
dereference, noticed by Jim Meyering.

Disks with a trailing digit in their path (eg /dev/loop0 or
/dev/dm0) have an extra 'p' appended before the partition
number (eg, to form /dev/loop0p1 not /dev/loop01). Fix the
partition lookup to append this extra 'p' when required

* src/storage/parthelper.c: Add a 'p' before partition
  number if required

Otherwise, a malicious packet could cause a DoS via spurious
out-of-memory failure.

* src/uml/uml_driver.c (umlMonitorCommand): Validate that incoming
data is reliable before using it to allocate/dereference memory.
Don't report bogus errno on short read.
Reported by Jim Meyering.

Report that libvirt was built without that driver instead of
trying to connect to a libvirtd, when we know that this is
going to fail.

Otherwise this will segfault if DISPLAY is not defined.

Otherwise this will segfault if PATH is not defined.

Reported by Emre Erenoglu

This patch adds two new parameters to the vol-create-as command:

 --backing-vol <volume-name-or-key-or-path>
 --backing-vol-format <format-of-backing-vol>

  virsh # vol-create-as guest_images_lvm snapvol1 5G --backing-vol \
              rhel6vm1lun1
  Vol snapvol1 created

  virsh # vol-create-as image_dir qcow2snap2 5G --format qcow2 \
              --backing-vol imagevol1.qcow2 \
              --backing-vol-format qcow2
  Vol qcow2snap2 created

Additionally, the virsh man page update fixes incorrect snapshot
parameters that were included in my prior bulk volume command patch.

On Fedora 13 with sufficient mingw32-* packages installed, running
./autobuild.sh failed to cross-compile to mingw because
mingw32-pthreads installed a broken <pthread.h>.  With that
issue fixed, the build still failed due to use of O_SYNC.
Meanwhile, recent .spec.in changes got out of sync.

* bootstrap.conf (gnulib_modules): Add fcntl-h, for O_SYNC.
* .gnulib: Update to latest, to work around buggy pthreads-win32
library.
* bootstrap: Import latest from gnulib.
* mingw32-libvirt.spec.in: Distribute new file.

* src/util/threads.c (includes) [WIN32]: On mingw, favor native
threading over pthreads-win32 library.
* src/util/thread.h [WIN32] Likewise.
Suggested by Daniel P. Berrange.

When a disk is on a root squashed NFS server, it may not be
possible to stat() the disk file in virCgroupAllowDevice.
The virStorageFileGetMeta method may also fail to extract
the parent backing store. Both of these errors have to be
ignored to avoid breaking NFS deployments

* src/qemu/qemu_driver.c: Ignore errors in cgroup setup to
   keep root squash NFS happy

With libvirt and virsh now being used for much more than Xen, this
patch removes the outdated reference to Xen in the file header.

Allow to specify a proxy to be used by libcurl.

Pass a struct containing the parameters instead of passing each
one individually. This make future extensions a bit simpler.

This patch also includes the new vol-pool command.

This patch adds a new "vol-pool" command to virsh, to round out the
identifier conversion functions for volumes in virsh.  Now it is
possible to work with volumes when starting from just a volume key
or volume path.

* tools/virsh.c (opts_create): Add --paused option.
(cmdCreate): Pass appropriate flag.
* tools/virsh.pod: Document it.

https://bugzilla.redhat.com/show_bug.cgi?id=589465

Some guests (eg with badly configured grub, or Windows' installation cd)
require quick response from the console user. That's why we have a
"launchPaused" option in vdsm.

To implement it via libvirt, we need to ask libvirt not to call
qemuMonitorStartCPUs() after starting qemu.  Calling virDomainStop
immediately after the domain is up is inherently raceful.

* src/qemu/qemu_driver.c (qemudStartVMDaemon): Add new parameter;
all callers adjusted.
(qemudDomainCreate): Implement support for new flag.

* include/libvirt/libvirt.h.in (virDomainCreateFlags): Add
VIR_DOMAIN_START_PAUSED.
* src/libvirt.c (virDomainCreateXML): Update documentation.
* src/lxc/lxc_driver.c (lxcDomainCreateAndStart): Reject new flag
as unimplemented.
* src/opennebula/one_driver.c (oneDomainCreateAndStart):
Likewise.
* src/openvz/openvz_driver.c (openvzDomainCreateXML): Likewise.
* src/phyp/phyp_driver.c (phypDomainCreateAndStart): Likewise.
* src/qemu/qemu_driver.c (qemudDomainCreate): Likewise.
* src/test/test_driver.c (testDomainCreateXML): Likewise.
* src/uml/uml_driver.c (umlDomainCreate): Likewise.
* src/vbox/vbox_tmpl.c (vboxDomainCreateXML): Likewise.
* src/xen/xend_internal.c (xenDaemonCreateXML): Likewise.
* src/xenapi/xenapi_driver.c (xenapiDomainCreateXML): Likewise.

* This patch is a modification of a patch submitted by Nigel Jones.
  It fixes several memory leaks on device addition/removal:

1. Free the virNodeDeviceDefPtr in udevAddOneDevice if the return
   value is non-zero

2. Always release the node device reference after the device has been
   processed.

* Refactored for better readability per the suggestion of clalance