Currently, qemuDomainGetXMLDesc and qemudDomainGetInfo check for
outstanding synchronous job before (eventual) monitor entering.
However, there can be already async job set, e.g. migration.

Thus, when libvirtd is restarted, it will know if a domain is supposed
to be killed or reset when it shuts down.

The previous patch removed all snapshots, but not the directory
where the snapshots lived, which is still a form of stale data.

* src/qemu/qemu_domain.c (qemuDomainRemoveInactive): Wipe any
snapshot directory.

This patch is mostly code motion - moving some functions out
of qemu_driver and into qemu_domain so they can be reused by
multiple qemu_* files (since qemu_driver.h must not grow).
It also adds a new helper function, qemuDomainRemoveInactive,
which will be used in the next patch.

* src/qemu/qemu_domain.h (qemuFindQemuImgBinary)
(qemuDomainSnapshotWriteMetadata, qemuDomainSnapshotForEachQcow2)
(qemuDomainSnapshotDiscard, qemuDomainSnapshotDiscardAll)
(qemuDomainRemoveInactive): New prototypes.
(struct qemu_snap_remove): New struct.
* src/qemu/qemu_domain.c (qemuDomainRemoveInactive)
(qemuDomainSnapshotDiscardAllMetadata): New functions.
(qemuFindQemuImgBinary, qemuDomainSnapshotWriteMetadata)
(qemuDomainSnapshotForEachQcow2, qemuDomainSnapshotDiscard)
(qemuDomainSnapshotDiscardAll): Move here...
* src/qemu/qemu_driver.c (qemuFindQemuImgBinary)
(qemuDomainSnapshotWriteMetadata, qemuDomainSnapshotForEachQcow2)
(qemuDomainSnapshotDiscard, qemuDomainSnapshotDiscardAll): ...from
here.
(qemuDomainUndefineFlags): Update caller.
* src/conf/domain_conf.c (virDomainRemoveInactive): Doc fixes.

The maximum bandwidth that can be consumed when migrating a domain
is better classified as an operational vs configuration parameter of
the dommain.  As such, store this parameter in qemuDomainObjPrivate
structure.

This patch creates an optional BeginJob queue size limit. When
active, all other attempts above level will fail. To set this
feature assign desired value to max_queued variable in qemu.conf.
Setting it to 0 turns it off.

Currently, we attempt to run sync job and async job at the same time. It
means that the monitor commands for two jobs can be run in any order.

In the function qemuDomainObjEnterMonitorInternal():
    if (priv->job.active == QEMU_JOB_NONE && priv->job.asyncJob) {
        if (qemuDomainObjBeginNestedJob(driver, obj) < 0)
We check whether the caller is an async job by priv->job.active and
priv->job.asynJob. But when an async job is running, and a sync job is
also running at the time of the check, then priv->job.active is not
QEMU_JOB_NONE. So we cannot check whether the caller is an async job
in the function qemuDomainObjEnterMonitorInternal(), and must instead
put the burden on the caller to tell us when an async command wants
to do a nested job.

Once the burden is on the caller, then only async monitor enters need
to worry about whether the VM is still running; for sync monitor enter,
the internal return is always 0, so lots of ignore_value can be dropped.

* src/qemu/THREADS.txt: Reflect new rules.
* src/qemu/qemu_domain.h (qemuDomainObjEnterMonitorAsync): New
prototype.
* src/qemu/qemu_process.h (qemuProcessStartCPUs)
(qemuProcessStopCPUs): Add parameter.
* src/qemu/qemu_migration.h (qemuMigrationToFile): Likewise.
(qemuMigrationWaitForCompletion): Make static.
* src/qemu/qemu_domain.c (qemuDomainObjEnterMonitorInternal): Add
parameter.
(qemuDomainObjEnterMonitorAsync): New function.
(qemuDomainObjEnterMonitor, qemuDomainObjEnterMonitorWithDriver):
Update callers.
* src/qemu/qemu_driver.c (qemuDomainSaveInternal)
(qemudDomainCoreDump, doCoreDump, processWatchdogEvent)
(qemudDomainSuspend, qemudDomainResume, qemuDomainSaveImageStartVM)
(qemuDomainSnapshotCreateActive, qemuDomainRevertToSnapshot):
Likewise.
* src/qemu/qemu_process.c (qemuProcessStopCPUs)
(qemuProcessFakeReboot, qemuProcessRecoverMigration)
(qemuProcessRecoverJob, qemuProcessStart): Likewise.
* src/qemu/qemu_migration.c (qemuMigrationToFile)
(qemuMigrationWaitForCompletion, qemuMigrationUpdateJobStatus)
(qemuMigrationJobStart, qemuDomainMigrateGraphicsRelocate)
(doNativeMigrate, doTunnelMigrate, qemuMigrationPerformJob)
(qemuMigrationPerformPhase, qemuMigrationFinish)
(qemuMigrationConfirm): Likewise.
* src/qemu/qemu_hotplug.c: Drop unneeded ignore_value.

qemuMigrationUpdateJobStatus (called in a loop by migration
and save tasks) uses qemuDomainObjEnterMonitorWithDriver;
however, that function ended up starting a nested job without
releasing the driver.

Since no one else is making nested calls, we can inline the
internal functions to properly track driver_locked.

* src/qemu/qemu_domain.h (qemuDomainObjBeginNestedJob)
(qemuDomainObjBeginNestedJobWithDriver)
(qemuDomainObjEndNestedJob): Drop unused prototypes.
* src/qemu/qemu_domain.c (qemuDomainObjEnterMonitorInternal):
Reflect driver lock to nested job.
(qemuDomainObjBeginNestedJob)
(qemuDomainObjBeginNestedJobWithDriver)
(qemuDomainObjEndNestedJob): Drop unused functions.

This doesn't abort migration job in any phase, yet.

Call qemu monitor command directly within a special job that is only
allowed during outgoing migration.

Call qemu monitor command directly within a special job that is only
allowed during outgoing migration.

Make MIGRATION_IN use the new helper methods.

This patch introduces several helper methods to deal with jobs and
phases during migration in a simpler manner.

In preparation for a future patch adding new virFile APIs.

* src/util/files.h, src/util/files.c: Move...
* src/util/virfile.h, src/util/virfile.c: ...here, and rename
functions to virFile prefix.  Macro names are intentionally
left alone.
* *.c: All '#include "files.h"' uses changed.
* src/Makefile.am (UTIL_SOURCES): Reflect rename.
* cfg.mk (exclude_file_name_regexp--sc_prohibit_close): Likewise.
* src/libvirt_private.syms: Likewise.
* docs/hacking.html.in: Likewise.
* HACKING: Regenerate.

* src/qemu/qemu_driver.c (qemudOpen, qemuDomainScreenshot)
(qemuDomainXMLFromNative, qemuDomainXMLToNative)
(qemudDomainBlockPeek, qemuCPUCompare, qemuCPUBaseline): Reject
unknown flags.
* src/qemu/qemu_migration.c (qemuMigrationConfirm): Likewise.
(_qemuMigrationCookie, qemuMigrationCookieXMLParse)
(qemuMigrationCookieXMLParseStr, qemuMigrationBakeCookie)
(qemuMigrationEatCookie): Make flags unsigned.
* src/qemu/qemu_domain.h: (qemuDomainDefFormatXML)
(qemuDomainFormatXML): Prefer unsigned flags.
* src/qemu/qemu_domain.c (qemuDomainDefFormatXML)
(qemuDomainFormatXML): Likewise.
(qemuDomainOpenLogHelper, qemuDomainCreateLog): Rename variable.

Asynchronous jobs may take long time to finish and may consist of
several phases which we need to now about to help with recovery/rollback
after libvirtd restarts.

Detect and react on situations when libvirtd was restarted or killed
when a job was active.

If libvirtd is restarted when a job is running, the new libvirtd process
needs to know about that to be able to recover and rollback the
operation.

Query commands are safe to be called during long running jobs (such as
migration). This patch makes them all work without the need to
special-case every single one of them.

The patch introduces new job.asyncCond condition and associated
job.asyncJob which are dedicated to asynchronous (from qemu monitor
point of view) jobs that can take arbitrarily long time to finish while
qemu monitor is still usable for other commands.

The existing job.active (and job.cond condition) is used all other
synchronous jobs (including the commands run during async job).

Locking schema is changed to use these two conditions. While asyncJob is
active, only allowed set of synchronous jobs is allowed (the set can be
different according to a particular asyncJob) so any method that
communicates to qemu monitor needs to check if it is allowed to be
executed during current asyncJob (if any). Once the check passes, the
method needs to normally acquire job.cond to ensure no other command is
running. Since domain object lock is released during that time, asyncJob
could have been started in the meantime so the method needs to recheck
the first condition. Then, normal jobs set job.active and asynchronous
jobs set job.asyncJob and optionally change the list of allowed job
groups.

Since asynchronous jobs only set job.asyncJob, other allowed commands
can still be run when domain object is unlocked (when communicating to
remote libvirtd or sleeping). To protect its own internal synchronous
commands, the asynchronous job needs to start a special nested job
before entering qemu monitor. The nested job doesn't check asyncJob, it
only acquires job.cond and sets job.active to block other jobs.

EnterMonitor and ExitMonitor methods are very similar to their
*WithDriver variants; consolidate them into EnterMonitorInternal and
ExitMonitorInternal to avoid (mainly future) code duplication.

This avoids code duplication and also avoids relying on good luck that
ignore_value(virDomainObjUnref(obj)) doesn't set errno.

Coverity warns if the majority of callers check a function for
errors, but a few don't; but in qemu_audit and qemu_domain, the
choice to not check for failures was safe.  In qemu_command, the
failure to generate a uuid can only occur on a bad pointer.

* src/qemu/qemu_audit.c (qemuAuditCgroup): Ignore failure to get
  cgroup controller.
* src/qemu/qemu_domain.c (qemuDomainObjEnterMonitor)
  (qemuDomainObjEnterMonitorWithDriver): Ignore failure to get
  timestamp.
* src/qemu/qemu_command.c (qemuParseCommandLine): Check for error.

The code emitting taint warnings was mistakenly thinking
that guests run from the QEMU session driver were tainted
for having high privileges. This is of course nonsense
since the session driver is always unprivileged

* src/qemu/qemu_domain.c: Don't warn for high privileges in
  non-privileged QEMU

* src/qemu/qemu_domain.c: (qemuDomainAppendLog)

  Free "message" in "cleanup".

Since we virEventRegisterDefaultImpl is now a public API, callers need
a way to invoke the default registered Handle and Timeout functions. We
already have general functions for these internally, so promote
them to the public API.

v2:
    Actually add APIs to libvirt.h

The QEMU integrates with the lock manager instructure in a number
of key places

 * During startup, a lock is acquired in between the fork & exec
 * During startup, the libvirtd process acquires a lock before
   setting file labelling
 * During shutdown, the libvirtd process acquires a lock
   before restoring file labelling
 * During hotplug, unplug & media change the libvirtd process
   holds a lock while setting/restoring labels

The main content lock is only ever held by the QEMU child process,
or libvirtd during VM shutdown. The rest of the operations only
require libvirtd to hold the metadata locks, relying on the active
QEMU still holding the content lock.

* src/qemu/qemu_conf.c, src/qemu/qemu_conf.h,
  src/qemu/libvirtd_qemu.aug, src/qemu/test_libvirtd_qemu.aug:
  Add config parameter for configuring lock managers
* src/qemu/qemu_driver.c: Add calls to the lock manager

Update the qemuDomainMigrateBegin method so that it accepts
an optional incoming XML document. This will be validated
for ABI compatibility against the current domain config,
and if this check passes, will be passed back out for use
by the qemuDomainMigratePrepare method on the target

* src/qemu/qemu_domain.c, src/qemu/qemu_domain.h,
  src/qemu/qemu_migration.c: Allow custom XML to be passed

Originally most of libvirt domain-specific calls were blocking
during a migration.
A new mechanism to allow specific calls (blkstat/blkinfo) to be
executed in such condition has been implemented.
In the long term it'd be desirable to get a more general
solution to mark further APIs as migration safe, without needing
special case code.

 * src/qemu/qemu_migration.c: add some additional job signal
   flags for doing blkstat/blkinfo during a migration
 * src/qemu/qemu_domain.c: add a condition variable that can be
   used to efficiently wait for the migration code to clear the
   signal flag
 * src/qemu/qemu_driver.c: execute blkstat/blkinfo using the
   job signal flags during migration

v2:
    Drop libvirt_private.syms changes

v3:
    Adjust for new virDomainEventStateNew argument

And update callers to actually respect the error

These VIR_XXXX0 APIs make us confused, use the non-0-suffix APIs instead.

How do these coversions works? The magic is using the gcc extension of ##.
When __VA_ARGS__ is empty, "##" will swallow the "," in "fmt," to
avoid compile error.

example: origin				after CPP
	high_level_api("%d", a_int)	low_level_api("%d", a_int)
	high_level_api("a  string")	low_level_api("a  string")

About 400 conversions.

8 special conversions:
VIR_XXXX0("") -> VIR_XXXX("msg") (avoid empty format) 2 conversions
VIR_XXXX0(string_literal_with_%) -> VIR_XXXX(%->%%) 0 conversions
VIR_XXXX0(non_string_literal) -> VIR_XXXX("%s", non_string_literal)
  (for security) 6 conversions
Signed-off-by: NLai Jiangshan <laijs@cn.fujitsu.com>

As well as taint warnings going to the main libvirt log,
add taint warnings to the per-domain logfile

  Domain id=3 is tainted: high-privileges
  Domain id=3 is tainted: disk-probing
  Domain id=3 is tainted: shell-scripts
  Domain id=3 is tainted: custom-monitor

* src/qemu/qemu_domain.c, src/qemu/qemu_domain.h: Enhance
  qemuDomainTaint to also log to the domain logfile
* src/qemu/qemu_driver.c: Pass -1 for logFD to taint methods to
  auto-append to logfile
* src/qemu/qemu_process.c: Pass open logFD at startup for taint
  methods

The qemuDomainAppendLog method allows writing a formatted string
to the end of the domain logfile, optionally opening it if needed.

* src/qemu/qemu_domain.c, src/qemu/qemu_domain.h: Add
  qemuDomainAppendLog

Move the qemuProcessLogReadFD and qemuProcessLogFD methods
into qemu_domain.c, renaming them to qemuDomainCreateLog
and qemuDomainOpenLog.

* src/qemu/qemu_domain.c, src/qemu/qemu_domain.h: Add
  qemuDomainCreateLog and qemuDomainOpenLog.
* src/qemu/qemu_process.c: Remove qemuProcessLogFD
  and qemuProcessLogReadFD

Wire up logging of VM tainting to the QEMU driver

 - If running QEMU as root user/group or without capabilities
   being cleared
 - If passing custom QEMU command line args
 - If issuing custom QEMU monitor commands
 - If using a network interface config with an associated
   shell script
 - If using a disk config relying on format probing

The warnings, per-VM appear in the main libvirtd logs

  11:56:17.571: 10832: warning : qemuDomainObjTaint:712 : Domain id=1 name='l2' uuid=c7a3edbd-edaf-9455-926a-d65c16db1802 is tainted: high-privileges
  11:56:17.571: 10832: warning : qemuDomainObjTaint:712 : Domain id=1 name='l2' uuid=c7a3edbd-edaf-9455-926a-d65c16db1802 is tainted: disk-probing

The taint flags are reset when the VM is stopped.

* src/qemu/qemu_domain.c, src/qemu/qemu_domain.h: Helper APIs
  for logging taint warnings
* src/qemu/qemu_driver.c: Log tainting with custom QEMU monitor
  commands and disk/net hotplug with unsupported configs
* src/qemu/qemu_process.c: Log tainting at startup based on
  unsupported configs

We already have virAsprintf, so picking a similar name helps for
seeing a similar purpose.  Furthermore, the prefix V before printf
generally implies 'va_list', even though this variant was '...', and
the old name got in the way of adding a new va_list version.

global rename performed with:

$ git grep -l virBufferVSprintf \
  | xargs -L1 sed -i 's/virBufferVSprintf/virBufferAsprintf/g'

then revert the changes in ChangeLog-old.