Update the ESX driver to use virReportError instead of
the ESX_ERROR & ESX_VI_ERROR custom macros
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

It was broken since forever as it expected a libxml2
XML_ELEMENT_NODE containing a XML_TEXT_NODE instead of
just a XML_TEXT_NODE.

This problem was not discovered for so long because
esxVI_String_Deserialize was not used until now.

Reported by Ata Bohra

Also ensure that the virBuffer used to store the downloaded data
does not overflow.

The two new APIs are rather trivial; based on bits and pieces of
other existing APIs.  But rather than blindly return 0 or 1 for
HasMetadata, I chose to first validate that the snapshot in
question in fact exists.

* src/esx/esx_driver.c (esxDomainSnapshotIsCurrent)
(esxDomainSnapshotHasMetadata): New functions.
* src/vbox/vbox_tmpl.c (vboxDomainSnapshotIsCurrent)
(vboxDomainSnapshotHasMetadata): Likewise.

Blindly returning success is misleading if the object no longer
exists; it is a bit better to check for existence up front before
returning information about that object.  This pattern matches the
fact that most of our other APIs check for existence as a side
effect prior to getting at the real piece of information being
queried.

* src/esx/esx_driver.c (esxDomainIsUpdated, esxDomainIsPersistent):
Add existence checks.
* src/vbox/vbox_tmpl.c (vboxDomainIsPersistent)
(vboxDomainIsUpdated): Likewise.

Appending an item to a list transfers ownership of that item to the
list owner. But an error can occur in between item allocation and
appending it to the list. In this case the item has to be freed
explicitly. This was not done in some special cases resulting in
possible memory leaks.

Reported by Coverity.

The current storage pools for NFS and iSCSI only require one host to
connect to. Future storage pools like RBD and Sheepdog will require
multiple hosts.

This patch allows multiple source hosts and rewrites the current
storage drivers.
Signed-off-by: NWido den Hollander <wido@widodh.nl>

Caused by commit 4445e16b that
made the code used the connection private data pointer before
it was initialized.

The code is splattered with a mix of

  sizeof foo
  sizeof (foo)
  sizeof(foo)

Standardize on sizeof(foo) and add a syntax check rule to
enforce it
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

When SASL requests auth credentials, try to look them up in the
config file first. If any are found, remove them from the list
that the user is prompted for
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Ensure that the functions in virauth.h have names matching the file
prefix, by renaming  virRequest{Username,Password} to
virAuthGet{Username,Password}
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

To follow latest naming conventions, rename src/util/authhelper.[ch]
to src/util/virauth.[ch].

* src/util/authhelper.[ch]: Rename to src/util/virauth.[ch]
* src/esx/esx_driver.c, src/hyperv/hyperv_driver.c,
  src/phyp/phyp_driver.c, src/xenapi/xenapi_driver.c: Update
  for renamed include files
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Convert drivers currently using the qparams APIs, to instead
use the virURIPtr query parameters directly.

* src/esx/esx_util.c, src/hyperv/hyperv_util.c,
  src/remote/remote_driver.c, src/xenapi/xenapi_utils.c: Remove
  use of qparams
* src/util/qparams.h, src/util/qparams.c: Delete
* src/Makefile.am, src/libvirt_private.syms: Remove qparams
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Instead of just typedef'ing the xmlURIPtr struct for virURIPtr,
use a custom libvirt struct. This allows us to fix various
problems with libxml2. This initially just fixes the query vs
query_raw handling problems.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Move error reporting out of the callers, into virURIParse
and virURIFormat, to get consistency.

* include/libvirt/virterror.h, src/util/virterror.c: Add VIR_FROM_URI
* src/util/viruri.c, src/util/viruri.h: Add error reporting
* src/esx/esx_driver.c, src/libvirt.c, src/libxl/libxl_driver.c,
  src/lxc/lxc_driver.c, src/openvz/openvz_driver.c,
  src/qemu/qemu_driver.c, src/qemu/qemu_migration.c,
  src/remote/remote_driver.c, src/uml/uml_driver.c,
  src/vbox/vbox_tmpl.c, src/vmx/vmx.c, src/xen/xen_driver.c,
  src/xen/xend_internal.c, tests/viruritest.c: Remove error
  reporting
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Since we defined a custom virURIPtr type, we should use a
virURIFree method instead of assuming it will always be
a typedef for xmlURIPtr

* src/util/viruri.c, src/util/viruri.h, src/libvirt_private.syms:
  Add a virURIFree method
* src/datatypes.c, src/esx/esx_driver.c, src/libvirt.c,
  src/qemu/qemu_migration.c, src/vmx/vmx.c, src/xen/xend_internal.c,
  tests/viruritest.c: s/xmlFreeURI/virURIFree/
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

On 64-bit platforms, unsigned long and unsigned long long are
identical, so we don't have to worry about overflow checks.
On 32-bit platforms, anywhere we narrow unsigned long long back
to unsigned long, we have to worry about overflow; it's easier
to do this in one place by having most of the code use the same
or wider types, and only doing the narrowing at the last minute.
Therefore, the memory set commands remain unsigned long, and
the memory get command now centralizes the overflow check into
libvirt.c, so that drivers don't have to repeat the work.

This also fixes a bug where xen returned the wrong value on
failure (most APIs return -1 on failure, but getMaxMemory
must return 0 on failure).

* src/driver.h (virDrvDomainGetMaxMemory): Use long long.
* src/libvirt.c (virDomainGetMaxMemory): Raise overflow.
* src/test/test_driver.c (testGetMaxMemory): Fix driver.
* src/rpc/gendispatch.pl (name_to_ProcName): Likewise.
* src/xen/xen_hypervisor.c (xenHypervisorGetMaxMemory): Likewise.
* src/xen/xen_driver.c (xenUnifiedDomainGetMaxMemory): Likewise.
* src/xen/xend_internal.c (xenDaemonDomainGetMaxMemory):
Likewise.
* src/xen/xend_internal.h (xenDaemonDomainGetMaxMemory):
Likewise.
* src/xen/xm_internal.c (xenXMDomainGetMaxMemory): Likewise.
* src/xen/xm_internal.h (xenXMDomainGetMaxMemory): Likewise.
* src/xen/xs_internal.c (xenStoreDomainGetMaxMemory): Likewise.
* src/xen/xs_internal.h (xenStoreDomainGetMaxMemory): Likewise.
* src/xenapi/xenapi_driver.c (xenapiDomainGetMaxMemory):
Likewise.
* src/esx/esx_driver.c (esxDomainGetMaxMemory): Likewise.
* src/libxl/libxl_driver.c (libxlDomainGetMaxMemory): Likewise.
* src/qemu/qemu_driver.c (qemudDomainGetMaxMemory): Likewise.
* src/lxc/lxc_driver.c (lxcDomainGetMaxMemory): Likewise.
* src/uml/uml_driver.c (umlDomainGetMaxMemory): Likewise.

Function xmlParseURI does not remove square brackets around IPv6
address when parsing. One of the solutions is making wrappers around
functions working with xmlURI*. This assures that uri->server will be
always properly assigned and it doesn't have to be changed when used
on some new place in the code.
For this purpose, functions virParseURI and virSaveURI were
added. These function are wrappers around xmlParseURI and xmlSaveUri
respectively.
Also there is one new syntax check function to prohibit these functions
anywhere else.

File changes:
 - src/util/viruri.h        -- declaration
 - src/util/viruri.c        -- definition
 - src/libvirt_private.syms -- symbol export
 - src/Makefile.am          -- added source and header files
 - cfg.mk                   -- added sc_prohibit_xmlURI
 - all others               -- ID name and include fixes

Adding "Expect:" to the header list stops libcurl from sending a
Expect header at all.

Before, a dummy Expect header was added that might confuse HTTP
proxies and result in HTTP error code 417 being reported.

Unlike .cvsignore under CVS, git allows for ignoring nested
names.  We weren't very consistent where new tests were
being ignored (some in .gitignore, some in tests/.gitignore),
and I found it easier to just consolidate everything.

* .gitignore: Subsume entries from subdirectories.
* daemon/.gitignore: Delete.
* docs/.gitignore: Likewise.
* docs/devhelp/.gitignore: Likewise.
* docs/html/.gitignore: Likewise.
* examples/dominfo/.gitignore: Likewise.
* examples/domsuspend/.gitignore: Likewise.
* examples/hellolibvirt/.gitignore: Likewise.
* examples/openauth/.gitignore: Likewise.
* examples/domain-events/events-c/.gitignore: Likewise.
* include/libvirt/.gitignore: Likewise.
* src/.gitignore: Likewise.
* src/esx/.gitignore: Likewise.
* tests/.gitignore: Likewise.
* tools/.gitignore: Likewise.

Add a new API virDomainShutdownFlags and define:

    VIR_DOMAIN_SHUTDOWN_DEFAULT        = 0,
    VIR_DOMAIN_SHUTDOWN_ACPI_POWER_BTN = (1 << 0),
    VIR_DOMAIN_SHUTDOWN_GUEST_AGENT    = (1 << 1),

Also define some flags for the reboot API

    VIR_DOMAIN_REBOOT_DEFAULT        = 0,
    VIR_DOMAIN_REBOOT_ACPI_POWER_BTN = (1 << 0),
    VIR_DOMAIN_REBOOT_GUEST_AGENT    = (1 << 1),

Although these two APIs currently have the same flags, using
separate enums allows them to expand separately in the future.

Add stub impls of the new API for all existing drivers

Reusing common code makes things smaller; it also buys us some
additional safety, such as now rejecting duplicate parameters
during a set operation.

* src/qemu/qemu_driver.c (qemuDomainSetBlkioParameters)
(qemuDomainSetMemoryParameters, qemuDomainSetNumaParameters)
(qemuSetSchedulerParametersFlags)
(qemuDomainSetInterfaceParameters, qemuDomainSetBlockIoTune)
(qemuDomainGetBlkioParameters, qemuDomainGetMemoryParameters)
(qemuDomainGetNumaParameters, qemuGetSchedulerParametersFlags)
(qemuDomainBlockStatsFlags, qemuDomainGetInterfaceParameters)
(qemuDomainGetBlockIoTune): Use new helpers.
* src/esx/esx_driver.c (esxDomainSetSchedulerParametersFlags)
(esxDomainSetMemoryParameters)
(esxDomainGetSchedulerParametersFlags)
(esxDomainGetMemoryParameters): Likewise.
* src/libxl/libxl_driver.c
(libxlDomainSetSchedulerParametersFlags)
(libxlDomainGetSchedulerParametersFlags): Likewise.
* src/lxc/lxc_driver.c (lxcDomainSetMemoryParameters)
(lxcSetSchedulerParametersFlags, lxcDomainSetBlkioParameters)
(lxcDomainGetMemoryParameters, lxcGetSchedulerParametersFlags)
(lxcDomainGetBlkioParameters): Likewise.
* src/test/test_driver.c (testDomainSetSchedulerParamsFlags)
(testDomainGetSchedulerParamsFlags): Likewise.
* src/xen/xen_hypervisor.c (xenHypervisorSetSchedulerParameters)
(xenHypervisorGetSchedulerParameters): Likewise.

https://bugzilla.redhat.com/show_bug.cgi?id=770954

https://bugzilla.redhat.com/show_bug.cgi?id=648855 mentioned a
misuse of 'an' where 'a' is proper; that has since been fixed,
but a search found other problems (some were a spelling error for
'and', while most were fixed by 'a').

* daemon/stream.c: Fix grammar.
* src/conf/domain_conf.c: Likewise.
* src/conf/domain_event.c: Likewise.
* src/esx/esx_driver.c: Likewise.
* src/esx/esx_vi.c: Likewise.
* src/rpc/virnetclient.c: Likewise.
* src/rpc/virnetserverprogram.c: Likewise.
* src/storage/storage_backend_fs.c: Likewise.
* src/util/conf.c: Likewise.
* src/util/dnsmasq.c: Likewise.
* src/util/iptables.c: Likewise.
* src/xen/xen_hypervisor.c: Likewise.
* src/xen/xend_internal.c: Likewise.
* src/xen/xs_internal.c: Likewise.
* tools/virsh.c: Likewise.

Many of these were mentioned by Yuri Chornoivan in:
https://bugzilla.redhat.com/show_bug.cgi?id=669506

* src/esx/esx_vi.c (esxVI_WaitForTaskCompletion): Fix spelling.
* src/conf/netdev_vport_profile_conf.c
(virNetDevVPortProfileParse): Likewise.
* src/xen/xend_internal.c (xenDaemonDomainSetVcpusFlags):
Likewise.
* src/xen/xm_internal.c (xenXMDomainSetVcpusFlags): Likewise.
* src/esx/esx_util.c (esxUtil_ResolveHostname): Likewise.
* src/storage/storage_backend_fs.c
(virStorageBackendFileSystemBuild): Likewise.
* daemon/libvirtd.conf: Likewise.
* src/util/logging.c (virLogMessage): Likewise.
* src/uml/uml_conf.c (umlBuildCommandLineNet): Likewise.
* src/vmx/vmx.c (virVMXFormatEthernet): Likewise.

The src/util/network.c file is a dumping ground for many different
APIs. Split it up into 5 pieces, along functional lines

 - src/util/virnetdevbandwidth.c: virNetDevBandwidth type & helper APIs
 - src/util/virnetdevvportprofile.c: virNetDevVPortProfile type & helper APIs
 - src/util/virsocketaddr.c: virSocketAddr and APIs
 - src/conf/netdev_bandwidth_conf.c: XML parsing / formatting
   for virNetDevBandwidth
 - src/conf/netdev_vport_profile_conf.c: XML parsing / formatting
   for virNetDevVPortProfile

* src/util/network.c, src/util/network.h: Split into 5 pieces
* src/conf/netdev_bandwidth_conf.c, src/conf/netdev_bandwidth_conf.h,
  src/conf/netdev_vport_profile_conf.c, src/conf/netdev_vport_profile_conf.h,
  src/util/virnetdevbandwidth.c, src/util/virnetdevbandwidth.h,
  src/util/virnetdevvportprofile.c, src/util/virnetdevvportprofile.h,
  src/util/virsocketaddr.c, src/util/virsocketaddr.h: New pieces
* daemon/libvirtd.h, daemon/remote.c, src/conf/domain_conf.c,
  src/conf/domain_conf.h, src/conf/network_conf.c,
  src/conf/network_conf.h, src/conf/nwfilter_conf.h,
  src/esx/esx_util.h, src/network/bridge_driver.c,
  src/qemu/qemu_conf.c, src/rpc/virnetsocket.c,
  src/rpc/virnetsocket.h, src/util/dnsmasq.h, src/util/interface.h,
  src/util/iptables.h, src/util/macvtap.c, src/util/macvtap.h,
  src/util/virnetdev.h, src/util/virnetdevtap.c,
  tools/virsh.c: Update include files

The default console type may vary based on the OS type. ie a Xen
paravirt guests wants a 'xen' console, while a fullvirt guests
wants a 'serial' console.

A plain integer default console type in the capabilities does
not suffice. Instead introduce a callback that is passed the
OS type.

* src/conf/capabilities.h: Use a callback for default console
  type
* src/conf/domain_conf.c, src/conf/domain_conf.h: Use callback
  for default console type. Add missing LXC/OpenVZ console types.
* src/esx/esx_driver.c, src/libxl/libxl_conf.c,
  src/lxc/lxc_conf.c, src/openvz/openvz_conf.c,
  src/phyp/phyp_driver.c, src/qemu/qemu_capabilities.c,
  src/uml/uml_conf.c, src/vbox/vbox_tmpl.c,
  src/vmware/vmware_conf.c, src/xen/xen_hypervisor.c,
  src/xenapi/xenapi_driver.c: Set default console type callback

Allow the user to call with nparams too small, per API documentation.

* src/esx/esx_driver.c (esxDomainGetMemoryParameters): Drop
redundant check.
(esxDomainGetSchedulerParametersFlags): Allow fewer than max.

Document the parameter names that will be used by
virDomain{Get,Set}SchedulerParameters{,Flags}, rather than
hard-coding those names in each driver, to match what is
done with memory, blkio, and blockstats parameters.

* include/libvirt/libvirt.h.in (VIR_DOMAIN_SCHEDULER_CPU_SHARES)
(VIR_DOMAIN_SCHEDULER_VCPU_PERIOD)
(VIR_DOMAIN_SCHEDULER_VCPU_QUOTA, VIR_DOMAIN_SCHEDULER_WEIGHT)
(VIR_DOMAIN_SCHEDULER_CAP, VIR_DOMAIN_SCHEDULER_RESERVATION)
(VIR_DOMAIN_SCHEDULER_LIMIT, VIR_DOMAIN_SCHEDULER_SHARES): New
field name macros.
* src/qemu/qemu_driver.c (qemuSetSchedulerParametersFlags)
(qemuGetSchedulerParametersFlags): Use new defines.
* src/test/test_driver.c (testDomainGetSchedulerParamsFlags)
(testDomainSetSchedulerParamsFlags): Likewise.
* src/xen/xen_hypervisor.c (xenHypervisorGetSchedulerParameters)
(xenHypervisorSetSchedulerParameters): Likewise.
* src/xen/xend_internal.c (xenDaemonGetSchedulerParameters)
(xenDaemonSetSchedulerParameters): Likewise.
* src/lxc/lxc_driver.c (lxcSetSchedulerParametersFlags)
(lxcGetSchedulerParametersFlags): Likewise.
* src/esx/esx_driver.c (esxDomainGetSchedulerParametersFlags)
(esxDomainSetSchedulerParametersFlags): Likewise.
* src/libxl/libxl_driver.c (libxlDomainGetSchedulerParametersFlags)
(libxlDomainSetSchedulerParametersFlags): Likewise.

Allow the datacenter and compute resource parts of the path
to be prefixed with folders. Therefore, the way the path is
parsed has changed. Before, it was split in 2 or 3 items and
the items' meanings were determined by their positions. Now
the path can have 2 or more items and the the vCenter server
is asked whether a folder, datacenter of compute resource
with the specified name exists at the current hierarchy level.

Before the datacenter and compute resource lookup automatically
traversed folders during lookup. This is logic got removed
and folders have to be specified explicitly.

The proper datacenter path including folders is now used when
accessing a datastore over HTTPS. This makes virsh dumpxml
and define work for datacenters in folders.

https://bugzilla.redhat.com/show_bug.cgi?id=732676

And virtual hardware version 8.

Relatively straight-forward filtering.

* src/esx/esx_vi.h (esxVI_GetNumberOfSnapshotTrees)
(esxVI_GetSnapshotTreeNames): Add parameter.
* src/esx/esx_vi.c (esxVI_GetNumberOfSnapshotTrees)
(esxVI_GetSnapshotTreeNames): Allow leaf filtering.
* src/esx/esx_driver.c (esxDomainSnapshotNum)
(esxDomainSnapshotListNames, esxDomainSnapshotNumChildren)
(esxDomainSnapshotListChildrenNames): Pass new flag through.

Coverity detected that the only way to get to the cleanup label
is if objectSpec had been successfully allocated, so the null
check was dead code.

* src/esx/esx_vi.c (esxVI_LookupObjectContentByType): Drop
redundant null check.

It was fairly trivial to return snapshot listing based on a
point in the hierarchy, rather than starting at all roots.

* src/esx/esx_driver.c (esxDomainSnapshotNumChildren)
(esxDomainSnapshotListChildrenNames): New functions.

I noticed a couple typos in recent commits, and fixed the remaining
instances of them.

* docs/internals/command.html.in: Fix spelling errors.
* include/libvirt/libvirt.h.in (virConnectDomainEventCallback):
Likewise.
* python/libvirt-override.py (virEventAddHandle): Likewise.
* src/lxc/lxc_container.c (lxcContainerChild): Likewise.
* src/util/hash.c (virHashCreateFull): Likewise.
* src/storage/storage_backend_logical.c
(virStorageBackendLogicalMakeVol): Likewise.
* src/esx/esx_driver.c (esxFormatVMXFileName): Likewise.
* src/vbox/vbox_tmpl.c (vboxIIDIsEqual_v3_x): Likewise.

No need to request the parent of a snapshot if we aren't going to use it.

* src/esx/esx_vi.c (esxVI_GetSnapshotTreeByName): Make parent
optional.
* src/esx/esx_driver.c (esxDomainSnapshotCreateXML)
(esxDomainSnapshotLookupByName, esxDomainRevertToSnapshot)
(esxDomainSnapshotDelete): Simplify accordingly.

Commit 9f5e53e2 introduced the ability to filter snapshots to
just roots, but it was never implemented for ESX until now.

* src/esx/esx_vi.h (esxVI_GetNumberOfSnapshotTrees)
(esxVI_GetSnapshotTreeNames): Add parameter.
* src/esx/esx_vi.c (esxVI_GetNumberOfSnapshotTrees)
(esxVI_GetSnapshotTreeNames): Allow choice of recursion or not.
* src/esx/esx_driver.c (esxDomainSnapshotNum)
(esxDomainSnapshotListNames): Use it to limit to roots.