Spawn the compressor ourselves, instead of requiring the shell.

* src/qemu/qemu_migration.c (qemuMigrationToFile): Spawn
compression helper process when needed.

SELinux labeling and cgroup ACLs aren't required if we hand a
pre-opened fd to qemu.  All the more reason to love fd: migration.

* src/qemu/qemu_migration.c (qemuMigrationToFile): Skip steps
that are irrelevant in fd migration.

This points out that core dumps (still) don't work for root-squash
NFS, since the fd is not opened correctly.  This patch should not
introduce any functionality change, it is just a refactoring to
avoid duplicated code.

* src/qemu/qemu_migration.h (qemuMigrationToFile): New prototype.
* src/qemu/qemu_migration.c (qemuMigrationToFile): New function.
* src/qemu/qemu_driver.c (qemudDomainSaveFlag, doCoreDump): Use
it.

Direct access to an open file is so much simpler than passing
everything through a pipe!

* src/qemu/qemu_driver.c (qemudOpenAsUID)
(qemudDomainSaveImageClose): Delete.
(qemudDomainSaveImageOpen): Rename...
(qemuDomainSaveImageOpen): ...and drop read_pid argument.  Use
virFileOpenAs instead of qemudOpenAsUID.
(qemudDomainSaveImageStartVM, qemudDomainRestore)
(qemudDomainObjRestore): Rename...
(qemuDomainSaveImageStartVM, qemuDomainRestore)
(qemDomainObjRestore): ...and simplify accordingly.
(qemudDomainObjStart, qemuDriver): Update callers.

* src/storage/storage_backend.c (createRawFileOpHook): Change
signature.
(struct createRawFileOpHookData): Delete unused struct.
(virStorageBackendCreateRaw): Adjust caller.
* src/qemu/qemu_driver.c (struct fileOpHookData): Delete unused
struct.
(qemudDomainSaveFileOpHook): Rename...
(qemuDomainSaveFileOpHook): ...and change signature.
(qemudDomainSaveFlag): Adjust caller.

This patch intentionally doesn't change indentation, in order to
make it easier to review the real changes.

* src/util/util.h (VIR_FILE_OP_RETURN_FD, virFileOperationHook):
Delete.
(virFileOperation): Rename...
(virFileOpenAs): ...and reduce parameters.
* src/util/util.c (virFileOperationNoFork, virFileOperation):
Rename and simplify.
* src/qemu/qemu_driver.c (qemudDomainSaveFlag): Adjust caller.
* src/storage/storage_backend.c (virStorageBackendCreateRaw):
Likewise.
* src/libvirt_private.syms: Reflect rename.

This makes root-squash NFS saves more efficient.

* src/qemu/qemu_driver.c (qemudDomainSaveFlag): Use new
virFileOperation flag to open fd only once.

This allows direct saves (no compression, no root-squash NFS) to use
the more efficient fd: migration, which in turn avoids a race where
qemu exec: migration can sometimes fail because qemu does a generic
waitpid() that conflicts with the pclose() used by exec:.  Further
patches will solve compression and root-squash NFS.

* src/qemu/qemu_driver.c (qemudDomainSaveFlag): Use new function
when there is no compression.

Latent bug introduced in commit 2d6a5819 (Aug 2009), but not exposed
until commit 1859939a (Jan 2011).  Basically, when virExec creates a
pipe, it always marks libvirt's side as cloexec.  If libvirt then
wants to hand that pipe to another child process, things work great if
the fd is dup2()'d onto stdin or stdout (as with stdin: or exec:
migration), but if the pipe is instead used as-is (such as with fd:
migration) then qemu sees EBADF because the fd was closed at exec().

This is a minimal fix for the problem at hand; it is slightly racy,
but no more racy than the rest of libvirt fd handling, including the
case of uncompressed save images.  A more invasive fix, but ultimately
safer at avoiding leaking unintended fds, would be to _always and
atomically_ open all fds as cloexec in libvirt (thanks to primitives
like open(O_CLOEXEC), pipe2(), accept4(), ...), then teach virExec to
clear that bit for all fds explicitly marked to be handed to the child
only after forking.

* src/qemu/qemu_command.c (qemuBuildCommandLine): Clear cloexec
flag.
* tests/qemuxml2argvtest.c (testCompareXMLToArgvFiles): Tweak test.

* src/qemu/qemu_process.c (qemuProcessStart, qemuProcessStop): Fix
typos.
* docs/hooks.html.in: Document 'prepare' and 'release' hooks.

Otherwise, if something like doStopVcpus fails after the first
restore, a second restore is attempted and throws a useless
warning.

* src/qemu/qemu_driver.c (qemudDomainSaveFlag): Avoid second
restore of state label.

Add missing open curly brace between function declaration of non-linux
variant of qemudDomainInterfaceStats() and its body.
Signed-off-by: NPhilipp Hahn <hahn@univention.de>

Hotpluging host usb device by text mode will fail, because the monitor
command 'device_add' outputs 'husb: using...' if it succeeds, but we
think the command should not output anything.
Signed-off-by: NWen Congyang <wency@cn.fujitsu.com>

Add the compiler attribute to ensure we don't introduce any more
ref bugs like were just patched in commit 9741f346, then explicitly
mark the remaining places in code that are safe.

* src/qemu/qemu_monitor.h (qemuMonitorUnref): Mark
ATTRIBUTE_RETURN_CHECK.
* src/conf/domain_conf.h (virDomainObjUnref): Likewise.
* src/conf/domain_conf.c (virDomainObjParseXML)
(virDomainLoadStatus): Fix offenders.
* src/openvz/openvz_conf.c (openvzLoadDomains): Likewise.
* src/vmware/vmware_conf.c (vmwareLoadDomains): Likewise.
* src/qemu/qemu_domain.c (qemuDomainObjBeginJob)
(qemuDomainObjBeginJobWithDriver)
(qemuDomainObjExitRemoteWithDriver): Likewise.
* src/qemu/qemu_monitor.c (QEMU_MONITOR_CALLBACK): Likewise.
Suggested by Daniel P. Berrange.

This simplifies several callers that were repeating checks already
guaranteed by util.c, and makes other callers more robust to now
reject directories.  remote_driver.c was over-strict - access(,R_OK)
is only needed to execute a script file; a binary only needs
access(,X_OK) (besides, it's unusual to see a file with x but not
r permissions, whether script or binary).

* cfg.mk (sc_prohibit_access_xok): New syntax-check rule.
(exclude_file_name_regexp--sc_prohibit_access_xok): Exempt one use.
* src/network/bridge_driver.c (networkStartRadvd): Fix offenders.
* src/qemu/qemu_capabilities.c (qemuCapsProbeMachineTypes)
(qemuCapsInitGuest, qemuCapsInit, qemuCapsExtractVersionInfo):
Likewise.
* src/remote/remote_driver.c (remoteFindDaemonPath): Likewise.
* src/uml/uml_driver.c (umlStartVMDaemon): Likewise.
* src/util/hooks.c (virHookCheck): Likewise.

Steps to reproduce this bug:
1. virsh attach-disk domain --source diskimage --target sdb --sourcetype file --driver qemu --subdriver qcow2
   error: Failed to attach disk
   error: operation failed: adding scsi-disk,bus=scsi0.0,scsi-id=1,drive=drive-scsi0-0-1,id=scsi0-0-1 device failed: Property 'scsi-disk.drive' can't find value 'drive-scsi0-0-1'
2. service libvirtd restart
   Stopping libvirtd daemon:                                  [  OK  ]
   Starting libvirtd daemon:                                  [  OK  ]
3. virsh attach-disk domain --source diskimage --target sdb --sourcetype file --driver qemu --subdriver raw
   error: Failed to attach disk
   error: operation failed: adding lsi,id=scsi0,bus=pci.0,addr=0x6 device failed: Duplicate ID 'scsi0' for device

The reason is that we create a new scsi controller but we do not update
/var/run/libvirt/qemu/domain.xml.
Signed-off-by: NWen Congyang <wency@cn.fujitsu.com>

Steps to reproduce this bug:
# cat usb.xml
<hostdev mode='subsystem' type='usb'>
  <source>
    <address bus='0x001' device='0x003'/>
  </source>
</hostdev>
# virsh attach-device vm1 usb.xml
error: Failed to attach device from usb.xml
error: server closed connection:

The reason of this bug is that we set data.cgroup to NULL, and this will cause
libvirtd crashed.
Signed-off-by: NWen Congyang <wency@cn.fujitsu.com>

A future patch will change reference counting idioms; consolidating
this pattern now makes the next patch smaller (touch only the new
macro rather than every caller).

* src/qemu/qemu_monitor.c (QEMU_MONITOR_CALLBACK): New helper.
(qemuMonitorGetDiskSecret, qemuMonitorEmitShutdown)
(qemuMonitorEmitReset, qemuMonitorEmitPowerdown)
(qemuMonitorEmitStop, qemuMonitorEmitRTCChange)
(qemuMonitorEmitWatchdog, qemuMonitorEmitIOError)
(qemuMonitorEmitGraphics): Use it to reduce duplication.

Enhance the QEMU migration monitoring loop, so that it can get
a signal to change migration speed on the fly

* src/qemu/qemu_domain.h: Add signal for changing speed on the fly
* src/qemu/qemu_driver.c: Wire up virDomainMigrateSetSpeed driver
* src/qemu/qemu_migration.c: Support signal for changing speed

It is possible to set a migration speed limit when starting
migration. This new API allows the speed limit to be changed
on the fly to adjust to changing conditions

* src/driver.h, src/libvirt.c, src/libvirt_public.syms,
  include/libvirt/libvirt.h.in: Add virDomainMigrateSetMaxSpeed
* src/esx/esx_driver.c, src/lxc/lxc_driver.c,
  src/opennebula/one_driver.c, src/openvz/openvz_driver.c,
  src/phyp/phyp_driver.c, src/qemu/qemu_driver.c,
  src/remote/remote_driver.c, src/test/test_driver.c,
  src/uml/uml_driver.c, src/vbox/vbox_tmpl.c,
  src/vmware/vmware_driver.c, src/xen/xen_driver.c,
  src/libxl/libxl_driver.c: Stub new API

fallback to HMP drive_add/drive_del commands if not found in QMP

Avoids calling text monitor methods when it is know they will not
succeed and also results in nicer error messages.

Fix for bug https://bugzilla.redhat.com/show_bug.cgi?id=618970

The "prepare" hook is called very early in the VM statup process
before device labeling, so that it can allocate ressources not
managed by libvirt, such as DRBD, or for instance create missing
bridges and vlan interfaces.
* src/util/hooks.c src/util/hooks.h: add definitions for new hooks
  VIR_HOOK_QEMU_OP_PREPARE and VIR_HOOK_QEMU_OP_RELEASE
* src/qemu/qemu_process.c: use them in qemuProcessStart and
  qemuProcessStop()

With only a single caller to these two monitor commands, I
didn't need to wrap a new WithFds version, but just change
the command itself.

* src/qemu/qemu_monitor.h (qemuMonitorAddNetdev)
(qemuMonitorAddHostNetwork): Add parameters.
* src/qemu/qemu_monitor.c (qemuMonitorAddNetdev)
(qemuMonitorAddHostNetwork): Add support for fd passing.
* src/qemu/qemu_hotplug.c (qemuDomainAttachNetDevice): Use it to
simplify code.

This is also a bug fix - on the error path, qemu_hotplug would
leave the configfd file leaked into qemu.  At least the next
attempt to hotplug a PCI device would reuse the same fdname,
and when the qemu getfd monitor command gets a new fd by the
same name as an earlier one, it closes the earlier one, so there
is no risk of qemu running out of fds.

* src/qemu/qemu_monitor.h (qemuMonitorAddDeviceWithFd): New
prototype.
* src/qemu/qemu_monitor.c (qemuMonitorAddDevice): Move guts...
(qemuMonitorAddDeviceWithFd): ...to new function, and add support
for fd passing.
* src/qemu/qemu_hotplug.c (qemuDomainAttachHostPciDevice): Use it
to simplify code.
Suggested by Daniel P. Berrange.

qemu_monitor was already returning -1 and setting errno to EINVAL
on any attempt to send an fd without a unix socket, but this was
a silent failure in the case of qemuDomainAttachHostPciDevice.
Meanwhile, qemuDomainAttachNetDevice was doing some sanity checking
for a better error message; it's better to consolidate that to a
central point in the API.

* src/qemu/qemu_hotplug.c (qemuDomainAttachNetDevice): Move sanity
checking...
* src/qemu/qemu_monitor.c (qemuMonitorSendFileHandle): ...into
central location.
Suggested by Chris Wright.

THREADS.txt states that the contents of vm should not be read or
modified while the vm lock is not held, but that the lock must not
be held while performing a monitor command.  This fixes all the
offenders that I could find.

* src/qemu/qemu_process.c (qemuProcessStartCPUs)
(qemuProcessInitPasswords, qemuProcessStart): Don't modify or
refer to vm state outside lock.
* src/qemu/qemu_driver.c (qemudDomainHotplugVcpus): Likewise.
* src/qemu/qemu_hotplug.c (qemuDomainChangeGraphicsPasswords):
Likewise.

Steps to reproduce this bug:
# cat test.sh
  #! /bin/bash -x
  virsh start domain
  sleep 5
  virsh qemu-monitor-command domain 'cpu_set 2 online' --hmp
# while true; do ./test.sh ; done

Then libvirtd will crash.

The reason is that:
we add a reference of obj when we open the monitor. We will reduce this
reference when we free the monitor.

If the reference of monitor is 0, we will free monitor automatically and
the reference of obj is reduced.

But in the function qemuDomainObjExitMonitorWithDriver(), we reduce this
reference again when the reference of monitor is 0.

It will cause the obj be freed in the function qemuDomainObjEndJob().

Then we start the domain again, and libvirtd will crash in the function
virDomainObjListSearchName(), because we pass a null pointer(obj->def->name)
to strcmp().
Signed-off-by: NWen Congyang <wency@cn.fujitsu.com>

This bug was reported by Shi Jin(jinzishuai@gmail.com):
=============
# virsh attach-disk RHEL6RC /var/lib/libvirt/images/test3.img vdb \
        --driver file --subdriver qcow2
Disk attached successfully

# virsh save RHEL6RC /var/lib/libvirt/images/memory.save
Domain RHEL6RC saved to /var/lib/libvirt/images/memory.save

# virsh restore /var/lib/libvirt/images/memory.save
error: Failed to restore domain from /var/lib/libvirt/images/memory.save
error: internal error unsupported driver name 'file'
       for disk '/var/lib/libvirt/images/test3.img'
=============

We check the driver name when we start or restore VM, but we do
not check it while attaching a disk. This adds the same check on disk
driverName used in qemuBuildCommandLine to qemudDomainAttachDevice.
Signed-off-by: NWen Congyang <wency@cn.fujitsu.com>

Steps to reproduce this bug:
# virsh qemu-monitor-command domain 'cpu_set 2 online' --hmp
The domain has 2 cpus, and we try to set the third cpu online.
The qemu crashes, and this command will hang.

The reason is that the refs is not 1 when we unwatch the monitor.
We lock the monitor, but we do not unlock it. So virCondWait()
will be blocked.
Signed-off-by: NWen Congyang <wency@cn.fujitsu.com>

* Correct the documentation for cgroup: the swap_hard_limit indicates
  mem+swap_hard_limit.
* Change cgroup private apis to: virCgroupGet/SetMemSwapHardLimit
Signed-off-by: NNikunj A. Dadhania <nikunj@linux.vnet.ibm.com>

* src/qemu/qemu_monitor.h (qemuMonitorMigrateToFd): New
prototype.
* src/qemu/qemu_monitor.c (qemuMonitorMigrateToFd): New function.

POSIX states about dd:

If the bs=expr operand is specified and no conversions other than
sync, noerror, or notrunc are requested, the data returned from each
input block shall be written as a separate output block; if the read
returns less than a full block and the sync conversion is not
specified, the resulting output block shall be the same size as the
input block. If the bs=expr operand is not specified, or a conversion
other than sync, noerror, or notrunc is requested, the input shall be
processed and collected into full-sized output blocks until the end of
the input is reached.

Since we aren't using conv=sync, there is no zero-padding, but our
use of bs= means that a short read results in a short write.  If
instead we use ibs= and obs=, then short reads are collected and dd
only has to do a single write, which can make dd more efficient.

* src/qemu/qemu_monitor.c (qemuMonitorMigrateToFile):
Avoid 'dd bs=', since it can cause short writes.

Fixes bug in commit acacced8

* src/qemu/qemu_command.c (qemuBuildCommandLine):
s/INVALID_ARG/CONFIG_UNSUPPORTED/.
Reported by Daniel P. Berrange.

* src/qemu/qemu_monitor_text.h (qemuMonitorTextMigrate): Declare
in place of individual monitor commands.
* src/qemu/qemu_monitor_json.h (qemuMonitorJSONMigrate): Likewise.
* src/qemu/qemu_monitor_text.c (qemuMonitorTextMigrateToHost)
(qemuMonitorTextMigrateToCommand, qemuMonitorTextMigrateToFile)
(qemuMonitorTextMigrateToUnix): Delete.
* src/qemu/qemu_monitor_json.c (qemuMonitorJSONMigrateToHost)
(qemuMonitorJSONMigrateToCommand, qemuMonitorJSONMigrateToFile)
(qemuMonitorJSONMigrateToUnix): Delete.
* src/qemu/qemu_monitor.c (qemuMonitorMigrateToHost)
(qemuMonitorMigrateToCommand, qemuMonitorMigrateToFile)
(qemuMonitorMigrateToUnix): Consolidate shared code.

Outgoing migration still uses a Unix socket and or exec netcat until
the next patch.

* src/qemu/qemu_migration.c (qemuMigrationPrepareTunnel):
Replace Unix socket with simpler pipe.
Suggested by Paolo Bonzini.

As perhaps other hypervisor drivers use different capacity units,
do the checking in qemu driver instead of in conf/domain_conf.c.