An upstream change to libvirt:

https://www.redhat.com/archives/libvir-list/2013-November/msg00400.html

results in the output starting with a single space rather than
at the beginning of the line - this change will account for that
with an appropriately placed '.lstrip()'.

Also, if fail the check for the existence of the base64_file, then
indicate that's what the failure is rather than what would be a
false message of "The secret value mismatch with result" because
the secret_decoded_string is empty.

virttest.utils_env: Check the whether the key is none before using it.

When get vm objects from Env object, the key of the data maybe None.
if the key is None, will raise error like:
   'NoneType' object has no attribute 'startswith'.
Signed-off-by: NYunping Zheng <yunzheng@redhat.com>

virttest.aexpect: FD leak fixes

Instead of having a flat structure of files

BASE_DIR/[filename]-[aexpect_instance]

Make the aexpect instance the parent dir of the files

BASE_DIR/[aexpect_instance]/filename

That makes it easy to see the number of open aexpect
controlled process at any given point in time, while
keeping information available.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Make the aexpect server able to log into a file. As it
is, we don't really have an easy way to tell what's going
on in there.

On later patches, we'll implement better ways to manage
files generated by aexpect, as well as debugging mechanisms.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

The clients will do that on close() anyway.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Checking up the qemu vm code, I noticed that none of the
uses of vm.process actually requires it to be an Expect
object. By using a Tail object, we obtain the same features
we've been using the code for, and we get rid of an extra
expect reader pipe, that ends up accumulating over the course
of the tests.

So make a function aexpect.run_tail(), nearly identical to
aexpect.run_bg(), and make vm.process to be a tail object.
With this change,  we further reduce the open number of open
fds from:

23 -> 25 -> 27 -> 29 -> 31 -> 33 -> 35 -> 37 -> 39 -> 40

to:

22 -> 23 -> 24 -> 25 -> 26 -> 26 -> 28 -> 29 -> 30 -> 31

If you guys can see any disadvantages of simplifying things
here, let me know.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

When the aexpect "tail" thread is requested to stop, take the chance
and close the filedescriptor being used by that thread. This makes up
for a big decrease in the number of open fds over the course of a
default virt test set. Before the patch, in 10 migrations tests
executed, the evolution of fds open is:

26 -> 31 -> 36 -> 41 -> 46 -> 51 -> 56 -> 61 -> 66 -> 71

After the patch, the evolution goes:

23 -> 25 -> 27 -> 29 -> 31 -> 33 -> 35 -> 37 -> 39 -> 40

That's quite a nice cleanup.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NChris Evich <cevich@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Environment RuntimeError fix

Add a utility method to wrap functions that modify the
contents of env.data. In order to avoid the boring
repetition of code that gets an environment save lock,
perform data change and then release the lock, put that
into the decorator and use it.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

There is a background tcpdump thread that records DHCP leases
in the env["address_cache"] dictionary. Sometimes, this thread
might write to this dictionary while the environment object is
being saved, triggering a RuntimeError in virt-test.

So the solution involves:

1) Creating a save lock, (a threading reentrant lock object),
that is acquired by the env.save() method when it wants to
save env.data to a pickle file, and for methods that alter
env.data.

2) Move the tcpdump thread to a env object attribute, rather
than one of the keys of env.data, given that the lock object
can't be pickled.

With the implementation of env.save_lock, we can also fix
the problem identified by the locking unittest. The base of
the solution was Pavel's idea, including a good deal of the
implementation, so kudos to him.
Signed-off-by: NPavel Hrdina <phrdina@redhat.com>
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Sometimes, while virt-test is saving Env to a file, the
tcpdump thread writes to the Env dictionary "address_cache",
generating a:

RuntimeError: dictionary changed size during iteration

exception. Of course, any thread that modifies env data
while saving is taking place is going to create this
problem. So I implemented a similar situation in the
Env unittest code, so we have a reproducer, that can
be later fixed to use new Env locking mechanisms.

So, in case it is not clear, this unittest is supposed
to fail right now, and after said locking mechanism
is implemented, the test will start passing.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

qemu.tests: Use qemu-kvm -M ? to get machine type for physical resource ...

virt-libvirt: Extend test-runner default tests

By default, the runner was running all variants of a single test.
This makes it harder to spot certain types of test-specific problems.
Instead, the runner should run a wider variety of tests, spreading
the variant choices across a range of underlying functionality.

Unfortunately, there's not a cleaner way to specify a wide range of
tests w/o simply listing them all.  However, the list can be read
in from a file, which will help keep the maintenance effort low. I
hard-coded list file path as relative to wherever the test runner
module is located, since there's no reason to over complicate it.

I expect this list will change over time, so this is only a reasonable
starting place. Included are about 80 libvirt tests from across the
board, with at least one positive and one negative test for each topic.
I attempted to pick different operational varients from each set so,
for example, it doesn't just exercize the same set of parameters
repeatidly.

Including the install and remove, they all complete on my intel i7 based
laptop in under 30min, which was found to be reasonable.
Signed-off-by: NChris Evich <cevich@redhat.com>

Find in utils_misc the cpu vendor and flag check code are duplicate.
Remove the duplicate one and update related scripts need to update.
Signed-off-by: NYiqiao Pu <ypu@redhat.com>

Add option '--log-level' to control log level for the top level log file

Virsh Snapshot-edit test

Signed-off-by: NChris Evich <cevich@redhat.com>

Fix duplication of 'type_name' property from base-class. In other
words, ``libvirt_xml/devices/base.py`` defines ``TypedDeviceBase``
which already handles setting up methods for ``get_type_name(),
set_type_name(), del_type_name()`` to work on device elements
``<device_name type='foobar'>``.  So there's no point in having a
subclass re-define accessor methods for the same ``type`` attribute.
Worse, it's using a python built-in (``type``) which encourages other
bad names down the stack.  I searched high and low for any usage of this
``type`` attribute and any calls to instance methods ``get_type(),
set_type(), and del_type()``  but found none.
Signed-off-by: NChris Evich <cevich@redhat.com>

Identified four unittests which were incorrectly
sharing state internally and externally.  In most
cases the fixes were simple: isolate test state
for every test via setup().  In the case of tests
needing to mock virsh functions, I took the extra
precautionary step of forcing them to first over-ride
ALL virsh functions/methods with an exception.
This way, any functions that should be mocked in
unittests but are not, will be immediatly clear :)
Signed-off-by: NChris Evich <cevich@redhat.com>

All entry points must import the common module.
Signed-off-by: NChris Evich <cevich@redhat.com>

The Virsh class automatically sucks in all callable
symbols found in the module and converts them into
methods.  However, an omission from the exclusion
list caused the VirshConnectBack class to be needlessly
exposed as a method (since classes are callable).
This has the potential to mess with unittests which
must mock Virsh methods, but at the same time may need
to use this class.  Adding it to the exclusion list
will prevent those problems.

Devices were not properly having the virsh instance
passed on to them.
Signed-off-by: NChris Evich <cevich@redhat.com>

There is a secondary set of propcan instances used behind the
scenes inside accessor factories.  Whenever they were trying
to retrieve non-existing data, or receive invalid input,
they toss a variaety of LibvirtXMLError exception sub-classes
by API design.  However, this doesn't jive with the standard
attribute/dictionary key-check method behavior.

The libvirt_xml API is defined such that all internally sourced
problems always throw a LibvirtXMLError exception or subclass.
However, since libvirt_base is also a propcan subclass, these
classes were get in the way of the regular python __contains__
and has_key() behavior.

This change makes LibvirtXML subclasses all conform to the python
dict and instance-attribute access convention.  Without it, one
would need to wrap the 'has_key()' call inside a try...except
LibvirtXMLError, which is unnatural.
Signed-off-by: NChris Evich <cevich@redhat.com>

Added ability to get/set/del '<model type=>' element + attribute.
Signed-off-by: NChris Evich <cevich@redhat.com>

Since PropcanBase is now fully pickle-able, there is no need
for custom string-conversion (which was used for pickling in virtnet).
Removed these special methods to just let normal python dict string
conversion take over.
Signed-off-by: NChris Evich <cevich@redhat.com>

Signed-off-by: NChris Evich <cevich@redhat.com>

Signed-off-by: NLi Yang <liyang.fnst@cn.fujitsu.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Modifying spice cfg file to use nettype = bridge

Signed-off-by: NYu Mingfei <yumingfei@cn.fujitsu.com>

virttest: Record the output from QMP monitor when failed to get the gree...

virttest: Update khugepaged test timeout

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Fix a typo in netperf result