* src/libvirt_private.syms: keep all symbols from network.h private

The current virDomainObjListPtr object stores domain objects in
an array. This means that to find a particular objects requires
O(n) time, and more critically acquiring O(n) mutex locks.

The new impl replaces the array with a virHashTable, keyed off
UUID. Finding a object based on UUID is now O(1) time, and only
requires a single mutex lock. Finding by name/id is unchanged
in complexity.

In changing this, all code which iterates over the array had
to be updated to use a hash table iterator function callback.
Several of the functions which were identically duplicating
across all drivers were pulled into domain_conf.c

* src/conf/domain_conf.h, src/conf/domain_conf.c: Change
  virDomainObjListPtr to use virHashTable. Add a initializer
  method virDomainObjListInit, and rename virDomainObjListFree
  to virDomainObjListDeinit, since its not actually freeing
  the container, only its contents. Also add some convenient
  methods virDomainObjListGetInactiveNames,
  virDomainObjListGetActiveIDs and virDomainObjListNumOfDomains
  which can be used to implement the correspondingly named
  public API entry points in drivers
* src/libvirt_private.syms: Export new methods from domain_conf.h
* src/lxc/lxc_driver.c, src/opennebula/one_driver.c,
  src/openvz/openvz_conf.c, src/openvz/openvz_driver.c,
  src/qemu/qemu_driver.c, src/test/test_driver.c,
  src/uml/uml_driver.c, src/vbox/vbox_tmpl.c: Update all code
  to deal with hash tables instead of arrays for domains

This will simplify adding FindPoolSources support to more pool backends in
the future (as well as the test driver).

Similar in theory to *AssignDef type functions, this duplicate functionality
will be used by an future FindPoolSources implementations.

This adds simple support for configuring a guest with a QEMU/KVM
virtual hardware watchdog device.

These will be used by the test driver, so move them to a shareable space.

When configuring logging settings, keep more information about the
output destination. Add accessors to retrieve the filter and output
settings in the original string form; this to be used to set up
environment for a child process that also logs.

* src/util/logging.[ch]: add virLogGetFilters and virLogGetOutputs
  accessors and modify the internals (including virLogDefineOutput())
  to save the data needed for the accessors

* src/util/util.[ch]: Add virFileAbsPath() function to ensure an
  absolute path for a potentially realtive path.
* src/libvirt_private.syms: add it in libvirt private symbols

The patch implements the missing memory control APIs for lxc, i.e.,
domainGetMaxMemory, domainSetMaxMemory, domainSetMemory, and improves
domainGetInfo to return proper amount of used memory via cgroup.

* src/libvirt_private.syms: Export virCgroupGetMemoryUsage
  and add missing virCgroupSetMemory
* src/lxc/lxc_driver.c: Implement missing memory functions
* src/util/cgroup.c, src/util/cgroup.h: Add the function
  to get used memory

Will be used by test driver node device implementation.
Signed-off-by: NCole Robinson <crobinso@redhat.com>

Implementation of tunnelled migration, using a Unix Domain Socket
on the qemu backend.  Note that this requires very new versions of
qemu (0.10.7 at least) in order to get the appropriate bugfixes.
Signed-off-by: NChris Lalancette <clalance@redhat.com>

* src/util/storage_file.c: add virStorageFileGetMetadata() so that
  the caller does not need to open the file

Finally, we get to the point of all this.

Move virStorageGetMetadataFromFD() to virStorageFileGetMetadataFromFD()
and move to src/util/storage_file.[ch]

There's no functional changes in this patch, just code movement

* src/storage/storage_backend_fs.c: move code from here ...

* src/util/storage_file.[ch]: ... to here

* src/libvirt_private.syms: export virStorageFileGetMetadataFromFD()

Rename virStorageVolFormatFileSystem to virStorageFileFormat and
move to src/util/storage_file.[ch]

* src/Makefile.am: add src/util/storage_file.[ch]

* src/conf/storage_conf.[ch]: move enum from here ...

* src/util/storage_file.[ch]: .. to here

* src/libvirt_private.syms: update To/FromString exports

* src/storage/storage_backend.c, src/storage/storage_backend_fs.c,
  src/vbox/vbox_tmpl.c: update for above changes

* include/libvirt/libvirt.h.in: Public API contract for
  virStreamPtr object
* src/libvirt_public.syms: Export data stream APIs
* src/libvirt_private.syms: Export internal helper APIs
* src/libvirt.c: Data stream API driver dispatch
* src/datatypes.h, src/datatypes.c: Internal helpers for virStreamPtr
  object
* src/driver.h: Define internal driver API for streams
* .x-sc_avoid_write: Ignore src/libvirt.c because it trips
  up on comments including write()
* python/Makefile.am: Add libvirt-override-virStream.py
* python/generator.py: Add rules for virStreamPtr class
* python/typewrappers.h, python/typewrappers.c: Wrapper
  for virStreamPtr
* docs/libvirt-api.xml, docs/libvirt-refs.xml: Regenerate
  with new APIs

* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h: Add new
  qemuMonitorMigrateToCommand() API
* src/qemu/qemu_driver.c: Switch over to using the
  qemuMonitorMigrateToCommand() API for core dumps and save
  to file APIs

* src/conf/domain_conf.c: Don't assume all virDomainObjPtr have
  a non-NULL monitor_chr field in virDomainObjFormat.
* src/lxc/lxc_driver.c: Implement suspend/resume driver APis
* src/util/cgroup.c, src/util/cgroup.h: Support the 'freezer'
  cgroup controller
* src/libvirt_private.syms: Export virCgroupSetFreezerState
  and virCgroupGetFreezerState

* src/bridge.c, src/bridge.h, src/buf.c, src/buf.h, src/cgroup.c,
  src/cgroup.h, src/conf.c, src/conf.h, src/event.c, src/event.h,
  src/hash.c, src/hash.h, src/hostusb.c, src/hostusb.h,
  src/iptables.c, src/iptables.h, src/logging.c, src/logging.h,
  src/memory.c, src/memory.h, src/pci.c, src/pci.h, src/qparams.c,
  src/qparams.h, src/stats_linux.c, src/stats_linux.h,
  src/threads-pthread.c, src/threads-pthread.h, src/threads-win32.c,
  src/threads-win32.h, src/threads.c, src/threads.h, src/util.c,
  src/util.h, src/uuid.c, src/uuid.h, src/virterror.c,
  src/virterror_internal.h, src/xml.c, src/xml.h: Move all files
  into src/util/
* daemon/Makefile.am: Add -Isrc/util/ to build flags
* src/Makefile.am: Add -Isrc/util/ to build flags and update for
  moved files
* src/libvirt_private.syms: Export cgroup APIs since they're now
  in util rather than linking directly to drivers
* src/xen/xs_internal.c: Disable bogus virEventRemoveHandle call
  when built under PROXY
* proxy/Makefile.am: Update for changed file locations. Remove
  bogus build of event.c
* tools/Makefile.am, tests/Makefile.am: Add -Isrc/util/ to build flags

* src/Makefile.am: Add gnulib.la for base64 functions
* src/libvirt_private.syms: Export virFileNameMatchesSuffix

If the <encryption format='qcow'> element does not specify a secret
during volume creation, generate a suitable secret and add it to the
<encryption> tag.  The caller can view the updated <encryption> tag
using virStorageVolGetXMLDesc().

Similarly, when <encryption format='default'/> is specified while
creating a qcow or qcow2-formatted volume, change the format to "qcow"
and generate a secret as described above.

* src/storage_encryption_conf.h (VIR_STORAGE_QCOW_PASSPHRASE_SIZE,
  virStorageGenerateQcowPasphrase),
  src/storage_encryption_conf.c (virStorageGenerateQcowPasphrase),
  src/libvirt_private.syms: Add virStorageGenerateQcowPasphrase().
* src/storage_backend.c (virStoragegenerateQcowEncryption,
  virStorageBackendCreateQemuImg): Generate a passphrase and
  <encryption> when creating a qcow-formatted encrypted volume and the
  user did not supply the information.

This implementation stores the secrets in an unencrypted text file,
for simplicity in implementation and debugging.

(Symmetric encryption, e.g. using gpgme, will not be difficult to add.
Because the TLS private key used by libvirtd is stored unencrypted,
encrypting the secrets file does not currently provide much additional
security.)

* include/libvirt/virterror.h, src/virterror.c (VIR_ERR_NO_SECRET): New
  error number.
* po/POTFILES.in, src/Makefile.am: Add secret_driver.
* bootstrap: Use gnulib's base64 module.
* src/secret_driver.c, src.secret_driver.h, src/libvirt_private.syms:
  Add local secret driver.
* qemud/qemud.c (qemudInitialize): Use the local secret driver.

Add a <secret> XML handling API, separate from the local driver, to
avoid manually generating XML in other parts of libvirt.

* src/secret_conf.c, src/secret_conf.h: New files.
* po/POTFILES.in, src/Makefile.am: Add secret_conf.

Remove the bogus dependancy between node_device.c & storage_backend.c
by moving the virWaitForDevices into util.h where it can be shared
safely

* src/storage_backend_disk.c, src/storage_backend_logical.c,
  src/storage_backend_mpath.c, src/storage_backend_scsi.c: Replace
  virStorageBackendWaitForDevices with virFileWaitForDevices
* src/storage_backend.c, src/storage_backend.h: Remove
  virStorageBackendWaitForDevices, virWaitForDevices
* src/util.h, src/util.c: Add virFileWaitForDevices
* configure.in: Move xmlrpc check further down after pkgconfig
  is detected
* src/Makefile.am: Add missing XMLRPC_CFLAGS/LIBS to opennebula
* src/libvirt_private.syms: Add many missing exports

* src/pci.h, src/pci.c: Helper for iterating over PCI device
  resource files
* src/libvirt_private.syms: Export pciDeviceFileIterate

* src/Makefile.am: Add usb.h and usb.h to libvirt_util.la
* src/libvirt_private.syms: Export symbols
* src/usb.c, src/usb.h: Helper APIs for USB host devices

* src/libvirt_private.syms: Add a missing comment for
  storage_encryption_conf.h entries

Add option to domain XML for

     <memoryBacking>
        <hugepages/>
     </memoryBacking>

* configure.in: Add check for mntent.h
* qemud/libvirtd_qemu.aug, qemud/test_libvirtd_qemu.aug, src/qemu.conf
  Add 'hugetlbfs_mount' config parameter
* src/qemu_conf.c, src/qemu_conf.h: Check for -mem-path flag in QEMU,
  and pass it when hugepages are requested.
  Load hugetlbfs_mount config parameter, search for mount if not given.
* src/qemu_driver.c: Free hugetlbfs_mount/path parameter in driver shutdown.
  Create directory for QEMU hugepage usage, chowning if required.
* docs/formatdomain.html.in: Document memoryBacking/hugepages elements
* docs/schemas/domain.rng: Add memoryBacking/hugepages elements to schema
* src/util.c, src/util.h, src/libvirt_private.syms: Add virFileFindMountPoint
  helper API
* tests/qemuhelptest.c: Add -mem-path constants
* tests/qemuxml2argvtest.c, tests/qemuxml2xmltest.c: Add tests for hugepage
  handling
* tests/qemuxml2argvdata/qemuxml2argv-hugepages.xml,
  tests/qemuxml2argvdata/qemuxml2argv-hugepages.args: Data files for
  hugepage tests

Calling qsort() on the disks array causes disk to be
unneccessarily re-ordered, potentially breaking the
ability to boot if the boot disk gets moved later in
the list. The new algorithm will insert a new disk as
far to the end of the list as possible, while being
ordered correctly wrt other disks on the same bus.

* src/domain_conf.c, src/domain_conf.h: Remove disk sorting
  routines. Add API to insert a disk into existing list at
  the optimal position, without resorting disks
* src/libvirt_private.syms: Export virDomainDiskInsert
* src/xend_internal.c, src/xm_internal.c: Remove calls to
  qsort, use virDomainDiskInsert instead.
* src/qemu_driver.c: Remove calls to qsort, use virDoaminDiskInsert
  instead. Fix reordering bugs when hotunplugging disks and
  networks. Fix memory leak in disk/net unplug

Define an <encryption> tag specifying volume encryption format and
format-depenedent parameters (e.g. passphrase, cipher name, key
length, key).

Currently the only defined parameter is a reference to a "secret"
(passphrase/key) managed using the virSecret* API.

Only the qcow/qcow2 encryption format, and a "default" format used to
let libvirt choose the format during volume creation, is currently
supported.

This patch does not add any users; the <encryption> tag is added in
the following patches to both volumes (to support encrypted volume
creation) and domains.

* docs/*.html: Re-generate
* docs/formatstorageencryption.html.in, docs/sitemap.html.in:
  Add page describing storage encryption data format
* docs/schemas/Makefile.am, docs/schemas/storageencryption.rng:
  Add RNG schema for storage encryption format
* po/POTFILES.in: Add src/storage_encryption_conf.c
* src/libvirt_private.syms: Export virStorageEncryption* functions
* src/storage_encryption_conf.h, src/storage_encryption_conf.c: Internal
  helper APIs for dealing with storage encryption format
* libvirt.spec.in, mingw32-libvirt.spec.in: Add storageencryption.rng
  RNG schema

* include/libvirt/virterror.h, src/virterror.c: Add VIR_WAR_NO_SECRET
* src/libvirt_private.syms, src/datatypes.h, src/datatypes.c: Type
  virSecret struct definition and helper APIs
* src/driver.h: Sub-driver API definitions for secrets
* src/libvirt.c: Define new sub-driver for secrets

The qemuPrepareHostDevices() and qemuDomainReAttachHostDevices()
functions are clutter with a bunch of calls to pciGetDevice() and
pciFreeDevice() obscuring the basic logic.

Add a pciDeviceList type and add a qemuGetPciHostDeviceList() function
to build a list from a domain definition. Use this in prepare/re-attach
fto simplify things and eliminate the multiple pciGetDevice calls.

This is especially useful because in the next patch we need to iterate
the hostdevs list a third time and we also need a list type for keeping
track of active devices.

* src/pci.[ch]: add pciDeviceList type and also a per-device 'managed'
  property

* src/libvirt_private.syms: export the new functions

* src/qemu_driver.c: add qemuGetPciHostDeviceList() and re-write
  qemuPrepareHostDevices() and qemuDomainReAttachHostDevices() to use it

It turns out that the previous attempt at this doesn't work well
in the case of hotplug. We need qemuCheckPciHostDevice() to
disallow the reset affecting devices already attach to the guest,
but we still need to avoid double locking the virDomainObjPtr.

This is all getting messy, I've a better idea.

This reverts commit 63188082 and
c106c8a1.

* src/qemu_driver.c, src/pci.[ch], src/xen_unified.c,
  src/libvirt_private.syms: revert a bunch of stuff.

If a PCI device reset causes other devices to be reset, allow it so long
as those other devices are note assigned to another active domain.

Note, we need to take the driver lock qemudNodeDeviceReset() because the
check function will iterate over the domain list.

* src/qemu_conf.c: add qemuCheckPciHostDevice() to iterate over active
  domains checking whether the affected device is assigned

* src/pci.[ch]: add pciDeviceEquals() helper

Re-factor the hostdev hotplug code so that we can easily add PCI
hostdev hotplug to qemudDomainAttachHostDevice().

* src/qemu_driver.c: rename qemudDomainAttachHostDevice() to
  qemudDomainAttachHostUsbDevice(); make qemudDomainAttachHostDevice()
  handle all hostdev types

* src/libvirt_private.syms: export a couple of hostdev related
  ToString() functions

* src/logging.c src/logging.h src/libvirt_private.syms:
  define new functions virLogSetFromEnv and virLogParseDefaultPriority
* qemud/qemud.c src/libvirt.c tests/eventtest.c: cleanup to use the
  unified functions

* qemud/qemud.c src/logging.[ch]: Similar as for general libvirt, don't
  convert high priority levels to debug level. Ignore LIBVIRT_LOG_FILTERS
  and LIBVIRT_LOG_OUTPUTS when they're set to the empty string, otherwise
  they can override a valid setting from the config file. Send all
  settings through the parser functions for validation, so that the
  existence of a bad setting doesn't nullify a good setting that should
  have applied -- particularly the default output. Keep the order of
  precedence consistent for all variables between the environment and
  the config file.  Warn when an invalid log level, filter, or output
  is ignored.
* src/libvirt_private.syms: export internally a few convenience functions

A subsequent commit will add a "canonical" field to this structure,
this patch basically just prepares the way for that.

The new type is added, along with virCapabilitiesAlloc/FreeMachines()
helpers and a whole bunch of code to make the transition.

One quirk is that virCapabilitiesAddGuestDomain() and
virCapabilitiesAddGuest() take ownership of the machine list rather
than duping it. This makes sense to avoid needless copying.

* src/capabilities.h: add the virCapsGuestMachine struct and use it
  in virCapsGuestDomainInfo, add prototypes for new functions and
  update the AddGuest() prototypes

* src/capabilities.c: add code for allocating and freeing the new
  type, change the machines parameter to AddGuest() etc.

* src/libvirt_private.syms: export the new helpers

* src/qemu_conf.c: update all the machine type code to use the new
  struct

* src/xen_internal.c: ditto

* tests/testutilsqemu.c: ditto

* src/qemu_driver.c:  Add driver methods qemuGetSchedulerType,
  qemuGetSchedulerParameters, qemuSetSchedulerParameters
* src/lxc_driver.c: Fix to use unsigned long long consistently
  for schedular parameters
* src/cgroup.h, src/cgroup.c: Fix cpu_shares to take unsigned
  long long
* src/util.c, src/util.h, src/libvirt_private.syms: Add a
  virStrToDouble helper
* src/virsh.c: Fix handling of --set arg to schedinfo command
  to honour the designated data type of each schedular tunable
  as declared by the driver

Allow the driver level cgroup to be managed explicitly by the
hypervisor drivers, in order to detect whether to enable or
disable cgroup support for domains. Provides better error
reporting of failures. Also allow for creation of cgroups for
unprivileged drivers if controller is accessible by the user.

* src/cgroup.c, src/cgroup.h: Add an API to obtain a driver cgroup
* src/lxc_conf.h, src/lxc_controller.c, src/lxc_driver.c:
  Obtain a driver cgroup at startup and use that instead of
  re-creating everytime.
* src/util.c, src/util.h, src/libvirt_private.syms: Add a
  virGetUserName() helper

Implement basic NIC hotplug support using the 'host_net_add' and
'pci_add' qemu monitor commands.

For now, we don't support 'bridge' or 'network' types.

Also, if pci_add fails, we currently fail to remove the backend
which we added.

Finally, NIC hot-unplug support is missing.

* src/qemu_driver.c: add qemudDomainAttachNetDevice()

* src/qemu_conf.[ch]: export qemuBuildNicStr(), qemuBuildHostNetStr()
  and qemuAssignNames()

* src/libvirt_private.syms: export virDomainNetTypeToString()