* src/rpc/virnetclient.c: Add debugging of ref counts

Commit 8665f855 introduced a slight regression in doc generation,
since make only quits a rule on the first failed command ending
with a newline rather than a semicolon.

* docs/Makefile.am (html/index.html): Don't use xmllint unless
xsltproc succeeded.
* .gitignore: Ignore recently updated stamp file name.

* src/util/event_poll.c: avoid file descriptors leak when
  virEventPollAddHandle fail on virEventPollInit function.

This reverts commit 40143fb6.

The patch prevents future compatibility if migration ever learns
how to do tunnelled without p2p.

If we get an I/O error in the async event callback for an RPC
client, we might not have consumed all pending data off the
wire. This could result in the callback being immediately
invoked again. At which point the same I/O might occur. And
we're invoked again. And again...And again...

Unregistering the async event callback if an error occurs is
a good safety net. The real error will be seen when the next
RPC method is invoked

* src/rpc/virnetclient.c: Unregister event callback on error

The current API build scripts will continue and exit with a zero
status even if they find problems. This has been the cause of many
build problems, or hidden build errors, in the past. Change the
scripts so they always exit with a non-zero status for any problems
they do not understand. Also turn off all debug output by default
so they respect $(AM_V_GEN)

* docs/Makefile.am: Use $(AM_V_GEN) for API/HTML scripts
* docs/apibuild.py, python/generator.py: Exit with non-zero status
  if problems are found. Also be silent, not outputting any debug
  messages.
* src/Makefile.am: Use $(AM_V_GEN) for ESX generator
* python/Makefile.am: Tweak rule

Caused by the new changed "dominfo" command.

* tools/virsh.c: new column "Managed save" for "cmdDominfo".
* tools/virsh.pod: Update document of "managedsave" to tell one can
  use "dominfo" to query whether a domain has any managed save image.

The problems:
    * Duplicate documents for "iface-name"
    * Lacks of document for "iface-mac"
    * Inconsistent option names with virsh help strings.

We can make the virsh migrate UI friendlier by supplying the
missing bit automatically instead of erroring out when requesting
--tunnelled without --p2p.

* tools/virsh.c (doMigrate): Make --p2p optional when using
--tunnelled.
* tools/virsh.pod (migrate): Tweak wording accordingly.

There were two API in driver.c that were silently masking flags
bits prior to calling out to the drivers, and several others
that were explicitly masking flags bits.  This is not
forward-compatible - if we ever have that many flags in the
future, then talking to an old server that masks out the
flags would be indistinguishable from talking to a new server
that can honor the flag.  In general, libvirt.c should forward
_all_ flags on to drivers, and only the drivers should reject
unknown flags.

In the case of virDrvSecretGetValue, the solution is to separate
the internal driver callback function to have two parameters
instead of one, with only one parameter affected by the public
API.  In the case of virDomainGetXMLDesc, it turns out that
no one was ever mixing VIR_DOMAIN_XML_INTERNAL_STATUS with
the dumpxml path in the first place; that internal flag was
only used in saving and restoring state files, which happened
to be in functions internal to a single file, so there is no
mixing of the internal flag with a public flags argument.
Additionally, virDomainMemoryStats passed a flags argument
over RPC, but not to the driver.

* src/driver.h (VIR_DOMAIN_XML_FLAGS_MASK)
(VIR_SECRET_GET_VALUE_FLAGS_MASK): Delete.
(virDrvSecretGetValue): Separate out internal flags.
(virDrvDomainMemoryStats): Provide missing flags argument.
* src/driver.c (verify): Drop unused check.
* src/conf/domain_conf.h (virDomainObjParseFile): Delete
declaration.
(virDomainXMLInternalFlags): Move...
* src/conf/domain_conf.c: ...here.  Delete redundant include.
(virDomainObjParseFile): Make static.
* src/libvirt.c (virDomainGetXMLDesc, virSecretGetValue): Update
clients.
(virDomainMemoryPeek, virInterfaceGetXMLDesc)
(virDomainMemoryStats, virDomainBlockPeek, virNetworkGetXMLDesc)
(virStoragePoolGetXMLDesc, virStorageVolGetXMLDesc)
(virNodeNumOfDevices, virNodeListDevices, virNWFilterGetXMLDesc):
Don't mask unknown flags.
* src/interface/netcf_driver.c (interfaceGetXMLDesc): Reject
unknown flags.
* src/secret/secret_driver.c (secretGetValue): Update clients.
* src/remote/remote_driver.c (remoteSecretGetValue)
(remoteDomainMemoryStats): Likewise.
* src/qemu/qemu_process.c (qemuProcessGetVolumeQcowPassphrase):
Likewise.
* src/qemu/qemu_driver.c (qemudDomainMemoryStats): Likewise.
* daemon/remote.c (remoteDispatchDomainMemoryStats): Likewise.

* tools/virsh.c: avoid double free of domain, when weight value of blkiotune
  less than 0, codes will free domain and jump to cleanup section, however,
  cleanup will free domain again.

Detected in valgrind run:

==21297== ERROR SUMMARY: 20 errors from 20 contexts (suppressed: 69 from 8)
==21297==
==21297== 1 errors in context 1 of 20:
==21297== Invalid read of size 4
==21297==    at 0x40E209B: virDomainFree (libvirt.c:2096)
==21297==    by 0x8065274: cmdBlkiotune (virsh.c:3695)
==21297==    by 0x8054CC1: vshCommandRun (virsh.c:13135)
==21297==    by 0x806B967: main (virsh.c:14487)
==21297==  Address 0x446ad48 is 0 bytes inside a block of size 36 free'd
==21297==    at 0x4005B0A: free (vg_replace_malloc.c:325)
==21297==    by 0x406814D: virFree (memory.c:310)
==21297==    by 0x40D6635: virReleaseDomain (datatypes.c:243)
==21297==    by 0x40D6C5E: virUnrefDomain (datatypes.c:280)
==21297==    by 0x40E20B9: virDomainFree (libvirt.c:2101)
==21297==    by 0x8065297: cmdBlkiotune (virsh.c:3613)
==21297==    by 0x8054CC1: vshCommandRun (virsh.c:13135)
==21297==    by 0x806B967: main (virsh.c:14487)
==21297==
==21297==
==21297== 1 errors in context 2 of 20:
==21297== Invalid read of size 4
==21297==    at 0x40E1FE6: virDomainFree (libvirt.c:2092)
==21297==    by 0x8065274: cmdBlkiotune (virsh.c:3695)
==21297==    by 0x8054CC1: vshCommandRun (virsh.c:13135)
==21297==    by 0x806B967: main (virsh.c:14487)
==21297==  Address 0x446ad48 is 0 bytes inside a block of size 36 free'd
==21297==    at 0x4005B0A: free (vg_replace_malloc.c:325)
==21297==    by 0x406814D: virFree (memory.c:310)
==21297==    by 0x40D6635: virReleaseDomain (datatypes.c:243)
==21297==    by 0x40D6C5E: virUnrefDomain (datatypes.c:280)
==21297==    by 0x40E20B9: virDomainFree (libvirt.c:2101)
==21297==    by 0x8065297: cmdBlkiotune (virsh.c:3613)
==21297==    by 0x8054CC1: vshCommandRun (virsh.c:13135)
==21297==    by 0x806B967: main (virsh.c:14487)

* how to reproduce?
  % valgrind -v --leak-check=full virsh blkiotune guestname --weight -1

"optional" is not a very good meta-syntactic construct in our man
page.  I scrubbed this, and additionally improved some documentation
on mutually exclusive options.  For example,

[[--live] [--config] | [--current]]

implies a set of optional flags, where within the set you can have
either --current or a choice of 0, 1, or both --live and --config.

* tools/virsh.pod: Use "[name]" rather than "optional name" for
optional arguments.

$@ already included $(srcdir)

When libvirtd restarts it will attempt to reconnect to existing
LXC containers. If it loads a XML state file for the container
the container will appear running. If we fail to read the PID
file, or fail to connect to the LXC monitor, we should be killing
off the guest, but if the VMs cgroup does not exist any more,
cleanup will get skipped. Reading the PID file is also pointless
since the PID is in the XML statefile

In lxcReconnectVM we do not need to read the PID file. If part
of the reconnect process fails we need to run the VM terminate
code as a safety net.

In lxcVMTerminate, if we can't obtain the VM cgroup, we know
the process has died, but we must still run lxcVMCleanup to
clear out the virDomainObjPtr live state

* src/lxc/lxc_driver.c: Fix cleanup of dead VMs on restart

We missed ending tag for paragraph element

These typos are introduced by file renaming in commit b17b4afa.

src/remote/qemu_protocol.x \
src/remote/remote_protocol.x \
src/rpc/gendispatch.pl:
    s/remote_generator/gendispatch/

src/rpc/genprotocol.pl:
    s/remote\/remote_protocol/remote_protocol/

The regression is introduced by Commit da1eba6b, the new
codes with this commit doesn't reset "ret" to "-1" when
it fails on parsing the device XML (live device attachment)

This patch changes the codes to reset the "ret" and "-1",
and also changes the codes so that it don't modify "ret"
for condition checking.

How to reproduce:

% cat test.xml
<disk type='oops' device='disk'>
  <driver name='qemu' type='raw'/>
  <source file='/var/lib/libvirt/images/test.img'/>
  <target dev='vda' bus='virtio'/>
</disk>

% virsh attach-device $domain test.xml
Device attached successfully

The device attachment failed actually with error "unknown disk type 'oops'",
however, it reports success.

As long as we guarantee RPC struct layout stability, we might as
well also guarantee RPC enum value constancy.

* src/Makefile.am (r1, r2, PDWTAGS): Adjust rule to pick up named
and anonymous enums.
* src/remote_protocol-structs: Add enum values.
* src/qemu_protocol-structs: Likewise.
* src/virnetprotocol-structs: Likewise.

Commit 13555416 introduced a nice feature without documenting it.

* src/libvirt.c (virDomainMigrate2): Add paragraph.

Enforce the recent flags cleanups - we want to use 'unsigned int flags'
in any of our APIs (except where backwards compatibility is important,
in the public migration APIs), and that all flags are checked for
validity (except when there are stub functions that completely
ignore the flags argument).

There are a few minor tweaks done here to avoid false positives:
signed arguments passed to open() are renamed oflags, and flags
arguments that are legitimately ignored are renamed flags_unused.

* cfg.mk (sc_flags_usage): New rule.
(exclude_file_name_regexp--sc_flags_usage): And a few exemptions.
(sc_flags_debug): Tweak wording.
* src/util/iohelper.c (runIO, main): Rename variable.
* src/util/util.c (virSetInherit): Likewise.
* src/fdstream.h (virFDStreamOpenFile, virFDStreamCreateFile):
Likewise.
* src/fdstream.c (virFDStreamOpenFileInternal)
(virFDStreamOpenFile, virFDStreamCreateFile): Likewise.
* src/util/command.c (virExecWithHook) [WIN32]: Likewise.
* src/util/util.c (virFileOpenAs, virDirCreate) [WIN32]: Likewise.
* src/locking/lock_manager.c (virLockManagerPluginNew)
[!HAVE_DLFCN_H]: Likewise.
* src/locking/lock_driver_nop.c (virLockManagerNopNew)
(virLockManagerNopAddResource, virLockManagerNopAcquire)
(virLockManagerNopRelease, virLockManagerNopInquire): Likewise.

Also fix a logic bug in xenXMDomain{Attach,Detach}DeviceFlags,
where (flags & VIR_DOMAIN_DEVICE_MODIFY_CURRENT) is always false.

* src/xen/xen_driver.c (xenUnifiedDomainXMLFromNative)
(xenUnifiedDomainXMLToNative, xenUnifiedDomainBlockPeek): Reject
unknown flags.
* src/xen/xen_hypervisor.c (xenHypervisorOpen)
(xenHypervisorGetDomainState): Likewise.
* src/xen/xen_inotify.c (xenInotifyOpen): Likewise.
* src/xen/xs_internal.c (xenStoreOpen, xenStoreDomainGetState)
(xenStoreDomainReboot): Likewise.
* src/xen/xend_internal.c (xenDaemonOpen, xenDaemonDomainReboot)
(xenDaemonDomainCoreDump, xenDaemonDomainGetState)
(xenDaemonDomainMigratePrepare, xenDaemonDomainSetVcpusFlags,
xenDaemonDomainGetVcpusFlags, xenDaemonAttachDeviceFlags,
xenDaemonDetachDeviceFlags): Likewise.
(xenDaemonDomainGetXMLDesc): Prefer unsigned flags.
* src/xen/xend_internal.h (xenDaemonDomainGetXMLDesc): Likewise.
* src/xen/xm_internal.h (xenXMDomainGetXMLDesc): Likewise.
* src/xen/xm_internal.c (xenXMDomainGetXMLDesc): Likewise.
(xenXMOpen, xenXMDomainGetState, xenXMDomainSetVcpusFlags)
(xenXMDomainGetVcpusFlags): Reject unknown flags.
(xenXMDomainAttachDeviceFlags, xenXMDomainDetachDeviceFlags):
Likewise, and avoid always-false conditional.
* src/xen/xen_driver.h (XEN_MIGRATION_FLAGS): New define.

Silently ignored flags get in the way of new features that
use those flags.

Regarding ESX migration flags - right now, ESX silently enforces
VIR_MIGRATE_PERSIST_DEST, VIR_MIGRATE_UNDEFINE_SOURCE, and
VIR_MIGRATE_LIVE, even if those flags were not supplied; it ignored
other flags.  This patch does not change the implied bits (it permits
but does not require them), but enforces only the supported bits.
If further cleanup is needed to be more particular about migration
flags, that should be a separate patch.

* src/esx/esx_device_monitor.c (esxDeviceOpen): Reject unknown
flags.
* src/esx/esx_driver.c (esxOpen, esxDomainReboot)
(esxDomainXMLFromNative, esxDomainXMLToNative)
(esxDomainMigratePrepare, esxDomainMigratePerform)
(esxDomainMigrateFinish): Likewise.
* src/esx/esx_interface_driver.c (esxInterfaceOpen): Likewise.
* src/esx/esx_network_driver.c (esxNetworkOpen): Likewise.
* src/esx/esx_nwfilter_driver.c (esxNWFilterOpen): Likewise.
* src/esx/esx_secret_driver.c (esxSecretOpen): Likewise.
* src/esx/esx_storage_driver.c (esxStorageOpen): Likewise.

The documentation for vshCommandOptString claims that it returns
-1 on a missing required argument, but in reality, that error
message was unreachable (it was buried inside an if clause that
is true only if the argument was present).  The code was so hairy
that I decided a rewrite would make it easier to understand,
and actually return the error values we want.

Meanwhile, our construction guarantees that all vshCmdOpt have
a non-null def member, so there are some redundant checks that
can be trimmed.

* tools/virsh.c (vshCommandOpt): Alter signature.
(vshCommandOptInt, vshCommandOptUInt, vshCommandOptUL)
(vshCommandOptString, vshCommandOptLongLong)
(vshCommandOptULongLong, vshCommandOptBool): Adjust all callers.
(vshCommandOptArgv): Remove dead condition.

Commit 461e0f1a broke migration, because there was a code path
that tried to enable an internal flag while still going through
the public function.  Split the internal flag into a separate
callback, and validate that flags do not overlap.

* src/conf/domain_conf.c (virDomainDefFormat): Split...
(virDomainDefFormatInternal): ...to separate the flag check.
(virDomainObjFormat): Adjust caller.

The "libvirt supports:" section on the main page of libvirt.org
contains a list of hypervisors with links that point to the sites of
the underlying virt technologies.  The entry for KVM points to
http://www.linux-kvm.org/, for example.  People coming to libvirt.org
for the first time are likely to know about those sites, and they're
probably interested in how libvirt manages those technologies.  This
patch points those links to the libvirt driver pages instead.  It also
consolidates KVM and QEMU as there is only one libvirt driver page for
them.  Finally, it adds a line about networking support.

v2: incorporate Eric's feedback adding project links to driver pages.

website: Add project links to KVM/QEMU driver page
website: Add project links to Xen driver page
website: Add project links to LXC driver page
website: Add project links to OpenVZ driver page
website: Add project links to UML driver page
website: Add project links to Virtualbox driver page
website: Add project links to ESX driver page
website: Add project links to VMware driver page

Commit f548480b broke migration v3 on qemu, because the driver
passed flags on through to qemu_migration even though
qemu_migration wasn't using those flags.

* src/qemu/qemu_migration.h (QEMU_MIGRATION_FLAGS): New define.
* src/qemu/qemu_driver.c: Simplify all migration callbacks.
* src/qemu/qemu_migration.c (qemuMigrationConfirm): Fix regression.

The previous patches only cleaned up ATTRIBUTE_UNUSED flags cases;
auditing the drivers found other places where flags was being used
but not validated.  In particular, domainGetXMLDesc had issues with
clients accepting a different set of flags than the common
virDomainDefFormat helper function.

* src/conf/domain_conf.c (virDomainDefFormat): Add common flag check.
* src/uml/uml_driver.c (umlDomainAttachDeviceFlags)
(umlDomainDetachDeviceFlags): Reject unknown
flags.
* src/vbox/vbox_tmpl.c (vboxDomainGetXMLDesc)
(vboxDomainAttachDeviceFlags)
(vboxDomainDetachDeviceFlags): Likewise.
* src/qemu/qemu_driver.c (qemudDomainMemoryPeek): Likewise.
(qemuDomainGetXMLDesc): Document common flag handling.
* src/libxl/libxl_driver.c (libxlDomainGetXMLDesc): Likewise.
* src/lxc/lxc_driver.c (lxcDomainGetXMLDesc): Likewise.
* src/openvz/openvz_driver.c (openvzDomainGetXMLDesc): Likewise.
* src/phyp/phyp_driver.c (phypDomainGetXMLDesc): Likewise.
* src/test/test_driver.c (testDomainGetXMLDesc): Likewise.
* src/vmware/vmware_driver.c (vmwareDomainGetXMLDesc): Likewise.
* src/xenapi/xenapi_driver.c (xenapiDomainGetXMLDesc): Likewise.

Our XML prefers "shareable" over "sharable".

* docs/internals/locking.html.in: s/sharable/shareable/
* src/locking/lock_driver_sanlock.c
(virLockManagerSanlockAddResource): Likewise.

This adds four options for virsh command attach-disk.

--cache option allows user to specify cache mode of disk device
from virsh command line when attaching a disk device.

--serial option allows user to specify serial string of disk device
from virsh command line when attaching a disk device.

--shareable option allows user to specify whether the disk device is
shareable between domains when attaching a disk device from virsh
command line.

--address option allows user to specify address of disk device when
attaching a disk device.

* src/rpc/virnettlscontext.c: s/read/write/

If the server succesfully validates the client cert, it will send
back a single byte, under TLS. If it fails, it will close the
connection. In this case, we were just reporting the standard
I/O error. The original RPC code had a special case hack for the
GNUTLS_E_UNEXPECTED_PACKET_LENGTH error code to make us report
a more useful error message

* src/rpc/virnetclient.c: Return ENOMSG if we get
  GNUTLS_E_UNEXPECTED_PACKET_LENGTH
* src/rpc/virnettlscontext.c: Report cert failure if we
  see ENOMSG

Many volume operations will fail if the volume in question is being
allocated.  These operations were returning VIR_ERR_INTERNAL_ERROR
when they should be returning VIR_ERR_OPERATION_INVALID.

This patch adds the --current option to "virsh setvcpus"
command. Currently "virsh setvcpus" command supports
"--live" and "--config" , but "--current" option.
From view of consistency, it's reasonable to support
"--current" option too.

When --current is specified, it affects a "current"
domain.
Signed-off-by: NTaku Izumi <izumi.taku@jp.fujitsu.com>

This patch extends qemudDomainSetVcpusFlags() function to support
VIR_DOMAIN_AFFECT_CURRENT flag.
Signed-off-by: NTaku Izumi <izumi.taku@jp.fujitsu.com>

This patch extends virDomainSetVcpusFlags API to support
VIR_DOMAIN_AFFECT_CURRENT flag.

Now because most APIs accept VIR_DOMAIN_AFFECT_CURRENT flags,
virDomainSetVcpusFlags API should also do.
Signed-off-by: NTaku Izumi <izumi.taku@jp.fujitsu.com>

* src/uml_conf.h: Add queue for dispatch of domain events
* src/uml_driver.c: Trigger domain events upon important lifecycle transitions

* docs/apps.html.in: Add EMOTIVE.
Reported by Alex Vaqué Brull <memfis@gmail.com>.
Signed-off-by: NEric Blake <eblake@redhat.com>

Rather than trying to clean up the ssh child ourselves, and risk
subtle differences from the socket creation error path, we can
just use the new APIs.

* src/rpc/virnetsocket.c (virNetSocketFree): Use new function.

By requesting the pid in virCommandRunAsync, fdstream was claiming
that it would manually wait for the process.  But on the failure
path, the child process was being leaked.

* src/fdstream.c (virFDStreamOpenFileInternal): Auto-reap child.