Several people have reported that if the .gnulib submodule is dirty,
then 'make' will go into an infinite loop attempting to rerun bootstrap,
because that never cleans up the dirty submodule.  By default, we
should halt and make the user investigate, but if the user doesn't
know why or care that the submodule is dirty, I also added the ability
to 'make CLEAN_SUBMODULE=1' to get things going again.

Also, while testing this, I noticed that when a submodule update was
needed, 'make' would first run autoreconf, then bootstrap (which
reruns autoreconf); adding a strategic dependency allows for less work.

* .gnulib: Update to latest, for maint.mk improvements.
* cfg.mk (_autogen): Also hook maint.mk, to run before autoreconf.
* autogen.sh (bootstrap): Refuse to run if gnulib is dirty, unless
user requests discarding gnulib changes.

Two more problems in "run.in" made the syntax-check fail.

I got an off-list report about a bad diagnostic:
Target network card mac 52:54:00:49:07:ccdoes not match source 52:54:00:49:07:b8

True to form, I've added a syntax check rule to prevent it
from recurring, and found several other offenders.

* cfg.mk (sc_require_whitespace_in_translation): New rule.
* src/conf/domain_conf.c (virDomainNetDefCheckABIStability): Add
space.
* src/esx/esx_util.c (esxUtil_ParseUri): Likewise.
* src/qemu/qemu_command.c (qemuCollectPCIAddress): Likewise.
* src/qemu/qemu_driver.c (qemuDomainSetMetadata)
(qemuDomainGetMetadata): Likewise.
* src/qemu/qemu_hotplug.c (qemuDomainChangeNetBridge): Likewise.
* src/rpc/virnettlscontext.c
(virNetTLSContextCheckCertDNWhitelist): Likewise.
* src/vmware/vmware_driver.c (vmwareDomainResume): Likewise.
* src/vbox/vbox_tmpl.c (vboxDomainGetXMLDesc, vboxAttachDrives):
Avoid false negatives.
* tools/virsh-domain.c (info_save_image_dumpxml): Reword.
Based on a report by Luwen Su.

To be able to test the QEMU monitor code, we need to have a fake
QEMU monitor server. This introduces a simple (dumb) framework
that can do this. The test case registers a series of items to
be sent back as replies to commands that will be executed. A
thread runs the event loop looking for incoming replies and
sending back this pre-registered data. This allows testing all
QEMU monitor code that deals with parsing responses and errors
from QEMU, without needing QEMU around
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Based on the similar gnulib commit 96ad9077.  The use of
$(_sc_search_regexp) already injects $(ME) into any output
messages, so a failure of these rules would look like this,
pre-patch:

maint.mk: maint.mk: use virStrToLong_*, not strtol variants

* cfg.mk (sc_prohibit_strncmp, sc_prohibit_strtol)
(sc_libvirt_unmarked_diagnostics): Drop redundant $(ME).

Yesterday's commit 15d2c9fa pointed out that virsh was still using
localtime(), which is not thread-safe, even though virsh is
definitely multi-threaded.  Even if we only ever triggered it from
one thread, it's better safe than sorry for maintenance purposes.

* cfg.mk (exclude_file_name_regexp--sc_prohibit_nonreentrant):
Tighten the rule.
* tools/virsh.c (vshOutputLogFile): Avoid localtime.
(vshEditWriteToTempFile, vshEditReadBackFile, cmdCd, cmdPwd)
(vshCloseLogFile): Avoid strerror.
* tools/console.c (vshMakeStdinRaw): Likewise.
* tools/virsh-domain.c (vshGenFileName): Fix spacing in previous
patch.

Nothing in the testsuite or examples directory should be translated,
as it is not part of the normally installed binary.  We already
meet this rule, but enforcing it will make it easier to remember.

Suggested by Daniel P. Berrange.

* cfg.mk (sc_prohibit_useless_translation): Enhance rule.

* cfg.mk: Whitelist virsh.h instead of virsh.c for strcasecmp check
* tools/virsh-domain.h, tools/virsh.h: Fix #define indentation

'make distcheck' was failing because a syntax check file,
.sc-start-sc_vulnerable_makefile_CVE-2012-3386, got left
behind.  I traced it to the 'distdir' rule depending on a
shortcut syntax-check name rather than the full rule name
normally used during 'local-check' from maint.mk.

* cfg.mk (distdir): Depend on full rule, not shorthand name.

* tools/virsh.c: New macro vshStrcasecmp
* tools/virsh-domain-monitor.c: Use vshStrcasecmp instead of
                                strcasecmp
* tools/virsh-snapshot.c: Likewise
* cfg.mk: Only avoid doing strcase checking for virsh.c

Make all the virNetClient* objects use virObject APIs for
reference counting
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Make all the virNetServer* objects use the virObject APIs
for reference counting
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Make virSocket use the virObject APIs for reference counting
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Make virNetSASLContext and virNetSASLSession use virObject APIs
for reference counting
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Make virNetTLSContext and virNetTLSSession use the virObject
APIs for reference counting
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

This introduces a fairly basic reference counted virObject type
and an associated virClass type, that use atomic operations for
ref counting.

In a global initializer (recommended to be invoked using the
virOnceInit API), a virClass type must be allocated for each
object type. This requires a class name, a "dispose" callback
which will be invoked to free memory associated with the object's
fields, and the size in bytes of the object struct.

eg,

   virClassPtr  connclass = virClassNew("virConnect",
                                        sizeof(virConnect),
                                        virConnectDispose);

The struct for the object, must include 'virObject' as its
first member

eg

  struct _virConnect {
    virObject object;

    virURIPtr uri;
  };

The 'dispose' callback is only responsible for freeing
fields in the object, not the object itself. eg a suitable
impl for the above struct would be

  void virConnectDispose(void *obj) {
     virConnectPtr conn = obj;
     virURIFree(conn->uri);
  }

There is no need to reset fields to 'NULL' or '0' in the
dispose callback, since the entire object will be memset
to 0, and the klass pointer & magic integer fields will
be poisoned with 0xDEADBEEF before being free()d

When creating an instance of an object, one needs simply
pass the virClassPtr eg

   virConnectPtr conn = virObjectNew(connclass);
   if (!conn)
      return NULL;
   conn->uri = virURIParse("foo:///bar")

Object references can be manipulated with

   virObjectRef(conn)
   virObjectUnref(conn)

The latter returns a true value, if the object has been
freed (ie its ref count hit zero)
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

The cfg.mk file rule to check for tab characters was not
applied to perl files. Much of our Perl code is full of
tabs as a result. Kill them, kill them all !

Commit f9ce7dad tried to kill uses of a raw street address, but
missed a few instances.  Automate things so we don't introduce
new problems in the future.

* cfg.mk (sc_copyright_address): New rule.
(exclude_file_name_regexp--sc_copyright_address): Add exemption.
* bootstrap.conf: Adjust offenders.
* build-aux/augeas-gentest.pl: Likewise.
* examples/systemtap/events.stp: Likewise.
* examples/systemtap/qemu-monitor.stp: Likewise.
* examples/systemtap/rpc-monitor.stp: Likewise.
* src/dtrace2systemtap.pl: Likewise.
* src/esx/esx_vi_generator.py: Likewise.
* src/hyperv/hyperv_wmi_generator.py: Likewise.
* src/remote/qemu_protocol.x: Likewise.
* src/remote/remote_protocol.x: Likewise.
* src/rpc/gensystemtap.pl: Likewise.
* src/rpc/virnetprotocol.x: Likewise.
* tests/object-locking.ml: Likewise.
* tools/virt-xml-validate.in: Likewise.

Any time we have a string with no % passed through gettext, a
translator can inject a % to cause a stack overread.  When there
is nothing to format, it's easier to ask for a string that cannot
be used as a formatter, by using a trivial "%s" format instead.

In the past, we have used --disable-nls to catch some of the
offenders, but that doesn't get run very often, and many more
uses have crept in.  Syntax check to the rescue!

The syntax check can catch uses such as
virReportError(code,
               _("split "
                 "string"));
by using a sed script to fold context lines into one pattern
space before checking for a string without %.

This patch is just mechanical insertion of %s; there are probably
several messages touched by this patch where we would be better
off giving the user more information than a fixed string.

* cfg.mk (sc_prohibit_diagnostic_without_format): New rule.
* src/datatypes.c (virUnrefConnect, virGetDomain)
(virUnrefDomain, virGetNetwork, virUnrefNetwork, virGetInterface)
(virUnrefInterface, virGetStoragePool, virUnrefStoragePool)
(virGetStorageVol, virUnrefStorageVol, virGetNodeDevice)
(virGetSecret, virUnrefSecret, virGetNWFilter, virUnrefNWFilter)
(virGetDomainSnapshot, virUnrefDomainSnapshot): Add %s wrapper.
* src/lxc/lxc_driver.c (lxcDomainSetBlkioParameters)
(lxcDomainGetBlkioParameters): Likewise.
* src/conf/domain_conf.c (virSecurityDeviceLabelDefParseXML)
(virDomainDiskDefParseXML, virDomainGraphicsDefParseXML):
Likewise.
* src/conf/network_conf.c (virNetworkDNSHostsDefParseXML)
(virNetworkDefParseXML): Likewise.
* src/conf/nwfilter_conf.c (virNWFilterIsValidChainName):
Likewise.
* src/conf/nwfilter_params.c (virNWFilterVarValueCreateSimple)
(virNWFilterVarAccessParse): Likewise.
* src/libvirt.c (virDomainSave, virDomainSaveFlags)
(virDomainRestore, virDomainRestoreFlags)
(virDomainSaveImageGetXMLDesc, virDomainSaveImageDefineXML)
(virDomainCoreDump, virDomainGetXMLDesc)
(virDomainMigrateVersion1, virDomainMigrateVersion2)
(virDomainMigrateVersion3, virDomainMigrate, virDomainMigrate2)
(virStreamSendAll, virStreamRecvAll)
(virDomainSnapshotGetXMLDesc): Likewise.
* src/nwfilter/nwfilter_dhcpsnoop.c (virNWFilterSnoopReqLeaseDel)
(virNWFilterDHCPSnoopReq): Likewise.
* src/openvz/openvz_driver.c (openvzUpdateDevice): Likewise.
* src/openvz/openvz_util.c (openvzKBPerPages): Likewise.
* src/qemu/qemu_cgroup.c (qemuSetupCgroup): Likewise.
* src/qemu/qemu_command.c (qemuBuildHubDevStr, qemuBuildChrChardevStr)
(qemuBuildCommandLine): Likewise.
* src/qemu/qemu_driver.c (qemuDomainGetPercpuStats): Likewise.
* src/qemu/qemu_hotplug.c (qemuDomainAttachNetDevice): Likewise.
* src/rpc/virnetsaslcontext.c (virNetSASLSessionGetIdentity):
Likewise.
* src/rpc/virnetsocket.c (virNetSocketNewConnectUNIX)
(virNetSocketSendFD, virNetSocketRecvFD): Likewise.
* src/storage/storage_backend_disk.c
(virStorageBackendDiskBuildPool): Likewise.
* src/storage/storage_backend_fs.c
(virStorageBackendFileSystemProbe)
(virStorageBackendFileSystemBuild): Likewise.
* src/storage/storage_backend_rbd.c
(virStorageBackendRBDOpenRADOSConn): Likewise.
* src/storage/storage_driver.c (storageVolumeResize): Likewise.
* src/test/test_driver.c (testInterfaceChangeBegin)
(testInterfaceChangeCommit, testInterfaceChangeRollback):
Likewise.
* src/vbox/vbox_tmpl.c (vboxListAllDomains): Likewise.
* src/xenxs/xen_sxpr.c (xenFormatSxprDisk, xenFormatSxpr):
Likewise.
* src/xenxs/xen_xm.c (xenXMConfigGetUUID, xenFormatXMDisk)
(xenFormatXM): Likewise.

We were defining 'func_or' as '|VIR_ERROR|...', which when put
inside 'func_re' resulted in a regex that matches everything in
isolation.  Thankfully, we always used func_re with a leading
anchor \<, and since the empty regex does not start a word, we
happened to get the result we wanted; but it's better to define
func_or without a leading space converted into a leading empty
alternation.

* cfg.mk (func_or): Strip leading space.

Pick up some build fixes in the latest gnulib.  In particular,
we want to ensure that official tarballs are secure, but don't
want to penalize people who don't run 'make dist', since fixed
automake still hasn't hit common platforms like Fedora 17.

* .gnulib: Update to latest, for Automake CVE-2012-3386 detection.
* bootstrap: Resync from gnulib.
* bootstrap.conf (gnulib_extra_files): Drop missing, since gnulib
has dropped it in favor of Automake's version.
* cfg.mk (local-checks-to-skip): Conditionally skip the security
check in cases where it doesn't matter.

Commands in node device group moved from virsh.c to virsh-nodedev.c,

* virsh.c: Remove commands in node device group.
* virsh-nodedev.c: New file, filled with commands in node device group
* po/POTFILES.in: Add virsh-nodedev.c
* cfg.mk: Skip to check config.h including for virsh-nodedev.c

Commands in host group moved from virsh.c to virsh-host.c,

* virsh.c: Remove commands in host group.
* virsh-host.c: New file, filled with commands in host group
* po/POTFILES.in: Add virsh-host.c
* cfg.mk: Skip to check config.h including for virsh-host.c

Commands to manage domain snapshot are moved from virsh.c to
virsh-snapshot.c.

* virsh.c: Remove domain snapshot commands.
* virsh-snapshot.c: New file, filled with domain snapshot commands.
* po/POTFILES.in: Add virsh-snapshot.c
* cfg.mk: Skip strcase and config.h including checking for
          virsh-snapshot.c

Commands to manage secret are moved from virsh.c to virsh-secret.c,
with a few helpers for secret command use.

* virsh.c: Remove secret commands and a few helpers.
           (vshCommandOptSecret, and vshCommandOptSecretBy)
* virsh-secret.c: New file, filled with secret commands and its helpers.
* po/POTFILES.in: Add virsh-secret.c
* cfg.mk: Skip to check config.h including for virsh-secret.c

Commands to manage network filter are moved from virsh.c to virsh-nwfilter.c,
with a few helpers for network filter command use.

* virsh.c: Remove network filter commands and a few helpers.
           (vshCommandOptNWFilter, and vshCommandOptNWFilterBy)
* virsh-nwfilter.c: New file, filled with network filter commands and its helpers.

* po/POTFILES.in: Add virsh-nwfilter.c

* cfg.mk: Skip to check config.h including for virsh-nwfilter.c

Commands to manage host interface are moved from virsh.c to
virsh-interface.c, with a few helpers for interface command use.

* virsh.c: Remove interface commands and a few helpers.
           (vshCommandOptInterface, vshCommandOptInterfaceBy)
* virsh-interface.c: New file, filled with interface commands and
                     its helpers.
* cfg.mk: Skip to check config.h including for virsh-interface.c
* po/POTFILES.in: Add virsh-interface.c

Commands to manage network are moved from virsh.c to virsh-network.c,
with a few helpers for network command use.

* virsh.c: Remove network commands and a few helpers.
* virsh-network.c: New file, filled with network commands and its
                   helpers.
* po/POTFILES.in: Add virsh-network.c
* cfg.mk: Skip to check config.h including for virsh-network.c

This splits commands of storage pool group into virsh-pool.c,
The helpers not for common use are moved too. Standard copyright
is added for the new file.

* tools/virsh.c:
  Remove commands for storage storage pool and a few helpers.
  (vshCommandOptVol, vshCommandOptVolBy).

* tools/virsh-pool.c:
  New file, filled with commands of storage pool group and its
  helpers.

* po/POTFILES.in:
  Add virsh-pool.c

* cfg.mk:
  Skip to check config.h including for virsh-pool.c

This splits commands of storage volume group into virsh-volume.c,
The helpers not for common use are moved too. Standard copyright
is added for the new file.

* tools/virsh.c:
  Remove commands for storage storage volume and a few helpers.
  (vshCommandOptVol, vshCommandOptVolBy).

* tools/virsh-volume.c:
  New file, filled with commands of storage volume group and its
  helpers.

* po/POTFILES.in:
  Add virsh-volume.c

* cfg.mk:
  Skip to check config.h including for virsh-volume.c

This splits commands to manage domain into virsh-domain.c,The helpers
not for common use are moved into them too. Standard copyright is added
for the new file.

* tools/virsh.c:
  - Remove commands for domain group, and one helper
    (vshDomainVcpuStateToString)
  - vshStreamSink is moved before commands's definition for it's
    also used by commands not of domain group, such as volUpload.

* tools/virsh-domain.c:
  - New file, commands for domain group and the one helper are
    moved into it.

* po/POTFILES.in:
  - Add virsh-domain.c

* cfg.mk:
  - Skip to check config.h including for virsh-domain.c

This splits commands commands to monitor domain status into
virsh-domain-monitor.c. The helpers not for common use are moved too.
Standard copyright is added.

* tools/virsh.c:
  - Remove commands for domain monitoring group and a few helpers (
    vshDomainIOErrorToString, vshGetDomainDescription,
    vshDomainControlStateToString, vshDomainStateToString) not for
    common use.
  - Remove (incldue "intprops.h").

* tools/virsh-domain-monitor.c:
  - New file, filled with commands of domain monitor group.
  - Add "intprops.h".

* cfg.mk:
  - Skip strcase checking for virsh-domain-monitor.c
  - Skip to check config.h including for virsh-domain-monitor.c

* po/POTFILES.in
  - Add virsh-domain-monitor.c

Update the legacy Xen drivers to use virReportError instead of
the statsError, virXenInotifyError, virXenStoreError,
virXendError, xenUnifiedError, xenXMError custom macros
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Update the VMX shared code to use virReportError instead of
the VMX_ERROR custom macro
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Update the libvirtd config handling code to use virReportError
instead of the virConfError custom macro
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Update the nodeinfo helper code to use virReportError instead
of the nodeReportError custom macro
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Update the security drivers to use virReportError instead of
the virSecurityReportError custom macro
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Update the Power-Hypervisor driver to use virReportError
instead of the PHYP_ERROR custom macro
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Update the Hyper-V driver to use virReportError instead
of the HYPERV_ERROR custom macro
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Update the ESX driver to use virReportError instead of
the ESX_ERROR & ESX_VI_ERROR custom macros
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>