In preparation for a future patch adding new virFile APIs.

* src/util/files.h, src/util/files.c: Move...
* src/util/virfile.h, src/util/virfile.c: ...here, and rename
functions to virFile prefix.  Macro names are intentionally
left alone.
* *.c: All '#include "files.h"' uses changed.
* src/Makefile.am (UTIL_SOURCES): Reflect rename.
* cfg.mk (exclude_file_name_regexp--sc_prohibit_close): Likewise.
* src/libvirt_private.syms: Likewise.
* docs/hacking.html.in: Likewise.
* HACKING: Regenerate.

Since libvirt is multi-threaded, we should use FD_CLOEXEC as much
as possible in the parent, and only relax fds to inherited after
forking, to avoid leaking an fd created in one thread to a fork
run in another thread.  This gets us closer to that ideal, by
making virCommand automatically clear FD_CLOEXEC on fds intended
for the child, as well as avoiding a window of time with non-cloexec
pipes created for capturing output.

* src/util/command.c (virExecWithHook): Use CLOEXEC in parent.  In
child, guarantee that all fds to pass to child are inheritable.
(getDevNull): Use CLOEXEC.
(prepareStdFd): New helper function.
(virCommandRun, virCommandRequireHandshake): Use pipe2.
* src/qemu/qemu_command.c (qemuBuildCommandLine): Simplify caller.

We already have a precedent of function documentation in C files,
where it is closer to the implementation (witness libvirt.h vs.
libvirt.c); maintaining docs in both files risks docs going stale.

While I was at it, I used consistent doxygen style on all comments.

* src/util/command.h: Remove duplicate docs, and move unique
documentation...
* src/util/command.c: ...here.
Suggested by Matthias Bolte.

The only 'void name(void)' style procedure in the protocol is 'close' that
is handled special, but also programming errors like a missing _args or
_ret suffix on the structs in the .x files can create such a situation by
accident. Making the generator aware of this avoids bogus errors from the
generator such as:

  Use of uninitialized value in exists at ./rpc/gendispatch.pl line 967.

Also this allows to get rid of the -c option and the special case code for
the 'close' procedure, as the generator handles it now correctly.

Reported by Michal Privoznik

It is common to see the sequence:

virErrorPtr save_err = virSaveLastError();
// do cleanup
virSetError(save_err);
virFreeError(save_err);

on cleanup paths.  But for functions where it is desirable to
return the errno that caused failure, this sequence can clobber
that errno.  virFreeError was already safe; this makes the other
two functions in the sequence safe as well, assuming all goes
well (on OOM, errno will be clobbered, but then again, save_err
won't reflect the real error that happened, so you are no longer
preserving the real situation - that's life with OOM).

* src/util/virterror.c (virSaveLastError, virSetError): Preserve
errno.

* src/qemu/qemu_driver.c: Fix uninitialized variable

This patch implements cfs_period and cfs_quota's modification.
We can use the command 'virsh schedinfo' to query or modify cfs_period and
cfs_quota.
If you query period or quota from config file, the value 0 means it does not set
in the config file.
If you set period or quota to config file, the value 0 means that delete current
setting from config file.
If you modify period or quota while vm is running, the value 0 means that use
current value.

This patch implements period and quota tunable XML configuration and parsing.
A quota or period of zero will be simply ignored.

This patch provides 4 APIs to get and set cpu.cfs_period_us and cpu.cfs_quota_us.

Introduce the function virCgroupForVcpu() to create sub directory for each vcpu.

qemu driver just accept xt_kbd codeset's keycode, so the lib virtkey
is used for translating keycodes from other codesets

Add virtkey lib for usage-improvment and keycode translating.
Add 4 internal API for the aim

const char *virKeycodeSetTypeToString(int codeset);
int virKeycodeSetTypeFromString(const char *name);
int virKeycodeValueFromString(virKeycodeSet codeset, const char *keyname);
int virKeycodeValueTranslate(virKeycodeSet from_codeset,
                             virKeycodeSet to_offset,
                             int key_value);

* include/libvirt/libvirt.h.in: extend virKeycodeSet enum
* src/Makefile.am: add new virtkeycode module and rule to generate
  virkeymaps.h
* src/util/virkeycode.c src/util/virkeycode.h: new module
* src/util/virkeycode-mapgen.py: python generator for virkeymaps.h
  out of keymaps.csv
* src/libvirt_private.syms: extend private symbols for new module
* .gitignore: add generated virkeymaps.h

Should keep it as the same as:
http://git.gnome.org/browse/gtk-vnc/commit/src/keymaps.csv

All master  keymaps are defined in a CSV file. THis covers
Linux keycodes, OSX keycodes, AT set1, 2 & 3, XT keycodes,
the XT encoding used by the Linux KBD driver, USB keycodes,
Win32 keycodes, the XT encoding used by Xorg on Cygwin,
the XT encoding used by Xorg on Linux with kbd driver.

* src/Makefile.am: added to EXTRA_DIST
* src/util/keymaps.csv: new file

* src/conf/domain_event.c (virDomainEventDispatch): Fix typo.
* src/internal.h (ATTRIBUTE_FMT_PRINTF): Likewise.
* src/libvirt.c (virStreamEventUpdateCallback): Likewise.
* src/remote/remote_driver.c (doRemoteOpen): Likewise.
* src/storage/storage_backend_fs.c (virStorageBackendProbeTarget):
Likewise.
* src/util/virterror.c (virConnCopyLastError, virCopyLastError):
Likewise.
* src/xen/xend_internal.h (xend_wait_for_devices): Likewise.

Though we prefer users to have SSH keys setup, virt-manager users still
depend on remote SSH connections to launch a password dialog. This fixes
launch ssh-askpass

Fix suggested by danpb

DMI table is Intel & Intel-compatible specific. Therefore other
architectures miss dmidecode command. So we always fail in searching
for that command on non-Intel architectures.

* src/rpc/virnetclient.c: Skip close attempt if virNetClientPtr
  is NULL

If a key purpose or usage field is marked as non-critical in the
certificate, then a data mismatch is not (ordinarily) a cause for
rejecting the connection

* src/rpc/virnettlscontext.c: Honour key usage/purpose criticality

If key usage or purpose data is not present in the cert, the
RFC recommends that access be allowed. Also fix checking of
key usage to include requirements for client/server certs,
and fix key purpose checking to treat data as a list of bits

* src/rpc/virnettlscontext.c: Fix mixed up error messages

DMI is Intel & Intel-compatible specific. Don't try to dump information
on non-compatible architectures, which results only in error message in
logs.

* src/libxl/libxl_driver.c (libxlDomainUndefineFlags): Use correct
enum value.
* src/remote_protocol-structs (remote_procedure): Likewise.

* src/libxl/libxl_driver.c: New callback for libxl_driver,
new function libxlDomainUndefineFlags, and changes libxlDomainUndefine
as a wrapper of libxlDomainUndefineFlags.

* src/qemu/qemu_driver.c: New call back for qemu_driver,
New function qemudDomainUndefineFlags, and changes on
qemudDomainUndefine.

This introduces a new API virDomainUndefineFlags to control the
domain undefine process, as the existing API virDomainUndefine
doesn't support flags.

Currently only flag VIR_DOMAIN_UNDEFINE_MANAGED_SAVE is supported.
If the domain has a managed save image, including
VIR_DOMAIN_UNDEFINE_MANAGED_SAVE in @flags will also remove that
file, and omitting the flag will cause undefine process to fail.

This patch also changes the behavior of virDomainUndefine, if the
domain has a managed save image, the undefine will be refused.

* cfg.mk (sc_copyright_format): Add to rule.
* src/util/files.h: Fix offenders.
* src/util/files.c: Likewise.

Gnutls requires that certificates have basic constraints present
to be used as a CA certificate. OpenSSL doesn't add this data
by default, so add a sanity check to catch this situation. Also
validate that the key usage and key purpose constraints contain
correct data

* src/rpc/virnettlscontext.c: Add sanity checking of certificate
  constraints

If the libvirt daemon or libvirt client is configured with bogus
certificates, it is very unhelpful to only find out about this
when a TLS connection is actually attempted. Not least because
the error messages you get back for failures are incredibly
obscure.

This adds some basic sanity checking of certificates at the
time the virNetTLSContext object is created. This is at libvirt
startup, or when creating a virNetClient instance.

This checks that the certificate expiry/start dates are valid
and that the certificate is actually signed by the CA that is
loaded.

* src/rpc/virnettlscontext.c: Add certificate sanity checks

Starting/ending jobs when closing the connection may reset any
error which was reported earlier in p2p migration. We must
save the original error before doing so. This means we can also
just call virConnectClose as normal, instead of virUnrefConnect

* src/qemu/qemu_migration.c: Preserve errors in p2p migration

Since the I/O callback registered against virNetSocket will
hold a reference on the virNetClient, we can't rely on the
virNetClientFree to be able to close the network connection.
The last reference will only go away when the event callback
fires (likely due to EOF from the server).

This is sub-optimal and can potentially cause a leak of the
virNetClient object if the server were to not explicitly
close the socket itself

* src/remote/remote_driver.c: Explicitly close the client
  object when disconnecting
* src/rpc/virnetclient.c, src/rpc/virnetclient.h: Add a
  virNetClientClose method

When unregistering an I/O callback from a virNetSocket object,
there is still a chance that an event may come in on the callback.
In this case it is possible that the virNetSocket might have been
freed already. Make use of a virFreeCallback when registering
the I/O callbacks and hold a reference for the entire time the
callback is set.

* src/rpc/virnetsocket.c: Register a free function for the
  file handle watch
* src/rpc/virnetsocket.h, src/rpc/virnetserverservice.c,
  src/rpc/virnetserverclient.c, src/rpc/virnetclient.c: Add
  a free function for the socket I/O watches

Remove the need for a virNetSocket object to be protected by
locks from the object using it, by introducing its own native
locking and reference counting

* src/rpc/virnetsocket.c: Add locking & reference counting

* src/rpc/virnetclient.c: Add debugging of ref counts

* src/util/event_poll.c: avoid file descriptors leak when
  virEventPollAddHandle fail on virEventPollInit function.

If we get an I/O error in the async event callback for an RPC
client, we might not have consumed all pending data off the
wire. This could result in the callback being immediately
invoked again. At which point the same I/O might occur. And
we're invoked again. And again...And again...

Unregistering the async event callback if an error occurs is
a good safety net. The real error will be seen when the next
RPC method is invoked

* src/rpc/virnetclient.c: Unregister event callback on error

The current API build scripts will continue and exit with a zero
status even if they find problems. This has been the cause of many
build problems, or hidden build errors, in the past. Change the
scripts so they always exit with a non-zero status for any problems
they do not understand. Also turn off all debug output by default
so they respect $(AM_V_GEN)

* docs/Makefile.am: Use $(AM_V_GEN) for API/HTML scripts
* docs/apibuild.py, python/generator.py: Exit with non-zero status
  if problems are found. Also be silent, not outputting any debug
  messages.
* src/Makefile.am: Use $(AM_V_GEN) for ESX generator
* python/Makefile.am: Tweak rule

There were two API in driver.c that were silently masking flags
bits prior to calling out to the drivers, and several others
that were explicitly masking flags bits.  This is not
forward-compatible - if we ever have that many flags in the
future, then talking to an old server that masks out the
flags would be indistinguishable from talking to a new server
that can honor the flag.  In general, libvirt.c should forward
_all_ flags on to drivers, and only the drivers should reject
unknown flags.

In the case of virDrvSecretGetValue, the solution is to separate
the internal driver callback function to have two parameters
instead of one, with only one parameter affected by the public
API.  In the case of virDomainGetXMLDesc, it turns out that
no one was ever mixing VIR_DOMAIN_XML_INTERNAL_STATUS with
the dumpxml path in the first place; that internal flag was
only used in saving and restoring state files, which happened
to be in functions internal to a single file, so there is no
mixing of the internal flag with a public flags argument.
Additionally, virDomainMemoryStats passed a flags argument
over RPC, but not to the driver.

* src/driver.h (VIR_DOMAIN_XML_FLAGS_MASK)
(VIR_SECRET_GET_VALUE_FLAGS_MASK): Delete.
(virDrvSecretGetValue): Separate out internal flags.
(virDrvDomainMemoryStats): Provide missing flags argument.
* src/driver.c (verify): Drop unused check.
* src/conf/domain_conf.h (virDomainObjParseFile): Delete
declaration.
(virDomainXMLInternalFlags): Move...
* src/conf/domain_conf.c: ...here.  Delete redundant include.
(virDomainObjParseFile): Make static.
* src/libvirt.c (virDomainGetXMLDesc, virSecretGetValue): Update
clients.
(virDomainMemoryPeek, virInterfaceGetXMLDesc)
(virDomainMemoryStats, virDomainBlockPeek, virNetworkGetXMLDesc)
(virStoragePoolGetXMLDesc, virStorageVolGetXMLDesc)
(virNodeNumOfDevices, virNodeListDevices, virNWFilterGetXMLDesc):
Don't mask unknown flags.
* src/interface/netcf_driver.c (interfaceGetXMLDesc): Reject
unknown flags.
* src/secret/secret_driver.c (secretGetValue): Update clients.
* src/remote/remote_driver.c (remoteSecretGetValue)
(remoteDomainMemoryStats): Likewise.
* src/qemu/qemu_process.c (qemuProcessGetVolumeQcowPassphrase):
Likewise.
* src/qemu/qemu_driver.c (qemudDomainMemoryStats): Likewise.
* daemon/remote.c (remoteDispatchDomainMemoryStats): Likewise.