As we did end with the 'next' branch, now PRs have to
be applied against the 'master' branch.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

virsh_secret_set_get: Prepare for upstream change

virt.test: modify vhost config for file_transfer test

Enlarge ksm overcommit test time

qemu.tests.pci_devices: Support for lspci verification

virsh_volume: Adjust for upstream change

Add some docstrings and other pylint fixes.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Additional check for (not only) correct pci addresses in qtree and lspci.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Split functions into subfunctions and prepare the test for multiple
parts with exception in the end.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The PCI address is hex so the correct regexp should match \w instead of \d
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

As the ksm size is calculate based on guest and host memory status,
when we run tests in big host, we need fill very big size of memory.
With current set up, it will failed with timeout to wait the results.
Signed-off-by: NYiqiao Pu <ypu@redhat.com>

image_name may be customized and not end with 64.
Signed-off-by: NYiqiao Pu <ypu@redhat.com>

platform is an old parameter we used. Should use vm_arch_name now.
Signed-off-by: NYiqiao Pu <ypu@redhat.com>

Currently for non-MSIX guest, qemu will disable vhost_net automatically,
even vhost is enabled. using vhostfoce=on to force enable vhost-net.
This patch modify the test configuration, only test "vhostforce=on"
on non-MSIX guest.
Signed-off-by: NYunping Zheng <yunzheng@redhat.com>

Upstream change adjusted the output of the virsh vol-list command
output to have a space in the front of the line, see:

https://www.redhat.com/archives/libvir-list/2013-November/msg00380.html

Add a strategic ".lstrip()" to the re.search() on the output

An upstream change to libvirt:

https://www.redhat.com/archives/libvir-list/2013-November/msg00400.html

results in the output starting with a single space rather than
at the beginning of the line - this change will account for that
with an appropriately placed '.lstrip()'.

Also, if fail the check for the existence of the base64_file, then
indicate that's what the failure is rather than what would be a
false message of "The secret value mismatch with result" because
the secret_decoded_string is empty.

virttest.utils_env: Check the whether the key is none before using it.

When get vm objects from Env object, the key of the data maybe None.
if the key is None, will raise error like:
   'NoneType' object has no attribute 'startswith'.
Signed-off-by: NYunping Zheng <yunzheng@redhat.com>

virttest.aexpect: FD leak fixes

Instead of having a flat structure of files

BASE_DIR/[filename]-[aexpect_instance]

Make the aexpect instance the parent dir of the files

BASE_DIR/[aexpect_instance]/filename

That makes it easy to see the number of open aexpect
controlled process at any given point in time, while
keeping information available.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Make the aexpect server able to log into a file. As it
is, we don't really have an easy way to tell what's going
on in there.

On later patches, we'll implement better ways to manage
files generated by aexpect, as well as debugging mechanisms.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

The clients will do that on close() anyway.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Checking up the qemu vm code, I noticed that none of the
uses of vm.process actually requires it to be an Expect
object. By using a Tail object, we obtain the same features
we've been using the code for, and we get rid of an extra
expect reader pipe, that ends up accumulating over the course
of the tests.

So make a function aexpect.run_tail(), nearly identical to
aexpect.run_bg(), and make vm.process to be a tail object.
With this change,  we further reduce the open number of open
fds from:

23 -> 25 -> 27 -> 29 -> 31 -> 33 -> 35 -> 37 -> 39 -> 40

to:

22 -> 23 -> 24 -> 25 -> 26 -> 26 -> 28 -> 29 -> 30 -> 31

If you guys can see any disadvantages of simplifying things
here, let me know.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

When the aexpect "tail" thread is requested to stop, take the chance
and close the filedescriptor being used by that thread. This makes up
for a big decrease in the number of open fds over the course of a
default virt test set. Before the patch, in 10 migrations tests
executed, the evolution of fds open is:

26 -> 31 -> 36 -> 41 -> 46 -> 51 -> 56 -> 61 -> 66 -> 71

After the patch, the evolution goes:

23 -> 25 -> 27 -> 29 -> 31 -> 33 -> 35 -> 37 -> 39 -> 40

That's quite a nice cleanup.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NChris Evich <cevich@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Environment RuntimeError fix

Add a utility method to wrap functions that modify the
contents of env.data. In order to avoid the boring
repetition of code that gets an environment save lock,
perform data change and then release the lock, put that
into the decorator and use it.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

There is a background tcpdump thread that records DHCP leases
in the env["address_cache"] dictionary. Sometimes, this thread
might write to this dictionary while the environment object is
being saved, triggering a RuntimeError in virt-test.

So the solution involves:

1) Creating a save lock, (a threading reentrant lock object),
that is acquired by the env.save() method when it wants to
save env.data to a pickle file, and for methods that alter
env.data.

2) Move the tcpdump thread to a env object attribute, rather
than one of the keys of env.data, given that the lock object
can't be pickled.

With the implementation of env.save_lock, we can also fix
the problem identified by the locking unittest. The base of
the solution was Pavel's idea, including a good deal of the
implementation, so kudos to him.
Signed-off-by: NPavel Hrdina <phrdina@redhat.com>
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Sometimes, while virt-test is saving Env to a file, the
tcpdump thread writes to the Env dictionary "address_cache",
generating a:

RuntimeError: dictionary changed size during iteration

exception. Of course, any thread that modifies env data
while saving is taking place is going to create this
problem. So I implemented a similar situation in the
Env unittest code, so we have a reproducer, that can
be later fixed to use new Env locking mechanisms.

So, in case it is not clear, this unittest is supposed
to fail right now, and after said locking mechanism
is implemented, the test will start passing.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

qemu.tests: Use qemu-kvm -M ? to get machine type for physical resource ...

virt-libvirt: Extend test-runner default tests

By default, the runner was running all variants of a single test.
This makes it harder to spot certain types of test-specific problems.
Instead, the runner should run a wider variety of tests, spreading
the variant choices across a range of underlying functionality.

Unfortunately, there's not a cleaner way to specify a wide range of
tests w/o simply listing them all.  However, the list can be read
in from a file, which will help keep the maintenance effort low. I
hard-coded list file path as relative to wherever the test runner
module is located, since there's no reason to over complicate it.

I expect this list will change over time, so this is only a reasonable
starting place. Included are about 80 libvirt tests from across the
board, with at least one positive and one negative test for each topic.
I attempted to pick different operational varients from each set so,
for example, it doesn't just exercize the same set of parameters
repeatidly.

Including the install and remove, they all complete on my intel i7 based
laptop in under 30min, which was found to be reasonable.
Signed-off-by: NChris Evich <cevich@redhat.com>

Find in utils_misc the cpu vendor and flag check code are duplicate.
Remove the duplicate one and update related scripts need to update.
Signed-off-by: NYiqiao Pu <ypu@redhat.com>

Add option '--log-level' to control log level for the top level log file

Virsh Snapshot-edit test

Signed-off-by: NChris Evich <cevich@redhat.com>

Fix duplication of 'type_name' property from base-class. In other
words, ``libvirt_xml/devices/base.py`` defines ``TypedDeviceBase``
which already handles setting up methods for ``get_type_name(),
set_type_name(), del_type_name()`` to work on device elements
``<device_name type='foobar'>``.  So there's no point in having a
subclass re-define accessor methods for the same ``type`` attribute.
Worse, it's using a python built-in (``type``) which encourages other
bad names down the stack.  I searched high and low for any usage of this
``type`` attribute and any calls to instance methods ``get_type(),
set_type(), and del_type()``  but found none.
Signed-off-by: NChris Evich <cevich@redhat.com>

Identified four unittests which were incorrectly
sharing state internally and externally.  In most
cases the fixes were simple: isolate test state
for every test via setup().  In the case of tests
needing to mock virsh functions, I took the extra
precautionary step of forcing them to first over-ride
ALL virsh functions/methods with an exception.
This way, any functions that should be mocked in
unittests but are not, will be immediatly clear :)
Signed-off-by: NChris Evich <cevich@redhat.com>

All entry points must import the common module.
Signed-off-by: NChris Evich <cevich@redhat.com>