The reboot function when rebooting back to a good kernel has a check
to make sure that a new kernel was indeed booted. But that check
uses a timeout value, which when calling the monitor will still
return success if the timeout is hit (no bug was found). It should
return an error to let the reboot code know that a new kernel was
not reached. Only the reboot code checks the return value of the
monitor.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

When doing a manual bisect, a build can fail or a test can be inconclusive.
In these cases it would be helpful to be able to skip the test entirely.

Link: http://lkml.kernel.org/r/1409164021-2136-1-git-send-email-chris.j.arges@canonical.comReviewed-by: NSatoru Takeuchi <satoru.takeuchi@gmail.com>
Signed-off-by: NChris J Arges <chris.j.arges@canonical.com>
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Add a way to run a patchcheck test on the commits that are in one branch
but not in another. This uses git cherry to find a list of commits to
test each one with.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

After the rewrite of the config bisect, the bisect manual was
removed. Add it back.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

After the rewrite of the config bisect, there were several unused
functions that can be removed.

One of the unused functions printed out the failed config nicer than
what the rewrite did, so I kept that and used it to output the
bad config.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

The new rewrite left out the CONFIG_BISECT_CHECK, which allows the
user to test that their "bad" config still is bad and their "good"
config still is good. This is especially important as the configs
are passed through a "make oldconfig" to update them with the lastest
kernel. Things could change that causes a bad config to work, or a
good config to break. The check is done after the configs have run
through the oldconfig processing.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

I never liked the way config-bisect worked. I would assume the bad config
had some config that broke the system. But it would not work if the bad
config just happened to be missing something that the good config had.

I rewrote the config-bisect to do this properly. It does a diff of the two
configs, and sets half of the configs that are in one and not the other.
The way it works is that when it "sets", it really just makes one copy
what the other has. That is, a "set" can be setting a:

  # CONFIG_FOO is not set

Basically, it looks at the differences between the two files and makes
them similar until it comes down to one config that makes it work or
not work depending on if it is set or not.

Note, if more than one config change makes the bad config not work, it
will only find one of them. But this is true with all bisect logic.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Some cleanup for improving readability as follows.

  - Initialize $ktest_config at its definition.
  - Put parentheses around the `config-file' argument in the usage message
    because it's a optional one.
  - Rename get_ktest_config{,s} to more descriptive get_mandatory_config{,s}.

Link: http://lkml.kernel.org/r/87fvmr30kb.wl%satoru.takeuchi@gmail.comSigned-off-by: NSatoru Takeuchi <satoru.takeuchi@gmail.com>
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

If we'd like to set the redirect target file of run_command(),
we should define $redirect before calling this function and should undef it
after calling this function. Since it's user-unfriendly, add 2nd parameter of
run_command() for this purpose.

Link: http://lkml.kernel.org/r/87vbvwokq8.wl%satoru.takeuchi@gmail.comSigned-off-by: NSatoru Takeuchi <satoru.takeuchi@gmail.com>
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

For those cases that it takes several tries to hit a bug, it would be
useful for ktest.pl to try a test multiple times before it considers
the test as a pass. To accomplish this, BISECT_TRIES ktest config
option has been added. It is default to one, as most of the time a
bisect only needs to try a test once. But the user can now up this
to make ktest run a given test multiple times. The first failure
that is detected will set a bisect bad. It only repeats on success.

Note, as with all race bugs, there's no guarantee that if it succeeds,
it is really a good bisect. But it helps in case the bug is somewhat
reliable.

You can set BISECT_TRIES to zero, and all tests will be considered
good, unless you also set BISECT_MANUAL.
Suggested-by: N"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

With the added variable ${KERNEL_VERSION}, it is useful to be
able to use parts of it for other variables.

For example, if you want to create a warnings file for each major
kernel version to test sub versions against you can create
your warnings file with like this:

  WARNINGS_FILE = warnings-file-${KERNEL_VERSION}

But this may add 3.8.12 or something, and we want all 3.8.* to
use the same file, and 3.10.* to use another file, and so on.
With the eval command we can, by adding:

  WARNINGS_FILE =~ s/(-file-\d+\.\d+).*/$1/

Which will chop off the extra characters after the 3.8.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Add a special variable that can be used in other variables called
${KERNEL_VERSION}. This will embed the current kernel version into
the variable. For example:

WARNINGS_FILE = ${OUTPUT_DIR}/warnings-${KERNEL_VERSION}

If the current version is v3.8 then the WARNINGS_FILE will become

  ${OUTPUT_DIR}/warnings-v3.8
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

 Currently ktest sends SIGINT to terminate the console.
 However, there are consoles which do not exit by this signal, for example,
 in my case, "virsh console <guest OS>". In such case, ktest is blocked in
 close_console(). It prevents this automate test.

This patch adds new option CLOSE_CONSOLE_SIGNAL which mean the
signal to terminate the console. Since its default value is "INT",
the original behavior isn't changed.

Link: http://lkml.kernel.org/r/87zjol8pl5.wl%satoru.takeuchi@gmail.comSigned-off-by: NSatoru Takeuchi <satoru.takeuchi@gmail.com>
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Correct spelling typo in various part of drivers
Signed-off-by: NMasanari Iida <standby24x7@gmail.com>
Signed-off-by: NJiri Kosina <jkosina@suse.cz>

Different tests may use a different machine. In such cases, we need to
try to get the current grub menu index. If the same grub menu is used
for two different machines, it may not be at the same index on the
second machine. A search for the index must be performed again.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

To save connecting and searching for a given grub menu for each test,
ktest.pl will cache the grub number it found. The problem is that
different tests might use a different grub menu, but ktest.pl will
ignore it.

Instead, have ktest.pl check if the grub menu it used to cache the
content is the same as when it grabbed the menu. If not, grab it again,
otherwise just return the cached value.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

The index of a line where a warning is tested can be returned
differently on different versions of gcc (or same version compiled
differently). That is, a tab + space can give different results. This
causes the warning check to produce a false positive. Removing the
index from the check fixes this issue.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

The reboot just wants to get to the next kernel. But if a warning (Call
Trace) appears, the monitor will report an error, and the reboot will
think something went wrong and power cycle the box, even though we
successfully made it to the next kernel.

Ignore warnings during the reboot until we get to the next kernel. It
will still timeout if we never get to the next kernel and then a power
cycle will happen. That's what we want it to do.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Sometimes when a test kernel passed fine, but on reboot it crashed,
ktest could get stuck and not proceed. This would be frustrating if you
let a test run overnight to find out the next morning that it was stuck
on the first test.

To fix this, I made reboot check for the REBOOT_SUCCESS_LINE. If the
line was not detected, then it would power cycle the box.

What it didn't cover was if the REBOOT_SUCCESS_LINE wasn't defined or if
a 'good' kernel did not display the line. Instead have it search for the
Linux banner "Linux version". The reboot just needs to get to the start
of the next kernel, it does not need to test if the next kernel makes it
to a boot prompt.

After we find the next kernel has booted, then we just wait for either
the REBOOT_SUCCESS_LINE to appear or the timeout.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Although the patchcheck test checks for warnings in the files that were
changed, this check does not catch warnings that were caused by header
file changes and the warnings appear in C files not touched by the
commit.

Add a new option called WARNINGS_FILE. If this option is set, then the
file it points to is read before bulid, and the file should contain a
list of known warnings. If a warning appears in the build, this file is
checked, and if the warning does not exist in this file, then it fails
the build showing the new warning.

If the WARNINGS_FILE points to a file that does not exist, this will
cause any warning in the build to fail.

A new test is also added called "make_warnings_file". This test will
create do a build and record any warnings it finds into the
WARNINGS_FILE. This test is something that can be run before other tests
to build a warnings file of "known warnings", ie, warnings that were
there before your changes.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Options are allowed to use other options, for example:

  LOG_FILE = ${OUTPUT_DIR}/${MACHINE}.log

where the option LOG_FILE used the options OUTPUT_DIR and MACHINE.

But if a test option were to use a default option, it will not get
substituted:

  OUTPUT_DIR = ${THIS_DIR}/${MACHINE}

  TEST_START
  OUTPUT_DIR = ${OUTPUT_DIR}/t1

For the above test, OUTPUT_DIR will stay literally "${OUTPUT_DIR}/t1"
and not be converted to "${THIS_DIR}/${MACHINE}/t1". When the test runs,
it will pass the ${OUTPUT_DIR} to the shell, which would probaly
interpret it as "", and the output directory will end up as "/t1".

Change the code where if a test option has its own option name in
its defined field, and a default option exists, then substitute the
default option in its place.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

The patchcheck test looks at what files are modified for each patch it
checks and makes sure that those files do not produce any warnings.

Unfortunately, when it read the diffstat, the newlines were added on the
files and this made compares miss warnings, and commits that should not
have passed, ktest let pass.

Fix this by using the perl command "chomp" that strips off whitespace at
the end of lines.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

If the user is doing a build or install bisect, there's no reason to
have them define CONSOLE, as the console does not need to be read. The
console only needs to be read for boot tests.

CONSOLE is not required for normal build or install tests, let's not
require it for bisect tests with BISECT_TYPE of build or install.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Sometimes a test kernel will crash or hang on reboot (this is even more
apparent when testing a config without CGROUPS on a box running
systemd). When this happens, on the next iteration of installing a
kernel, ktest will fail when it tries to install.

Have ktest do a check to see if the target can be connected to via ssh
before it tries to install. If it can't connect, then reboot again.
This time the reboot will fail because it can't connect and will force a
power cycle.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Commit fb16d891 "kconfig: replace 'oldnoconfig' with 'olddefconfig', and
keep the old name", changed ktest's default config update from
oldnoconfig to olddefconfig without adding oldnoconfig as a backup.
The make oldnoconfig works much better than its backup of:
   yes '' | make oldconfig

But due to this change, and the fact that ktest is used to build lots of
older kernels (and for bisects), it forgoes the oldnoconfig completely.

Cc: Adam Lee <adam8157@gmail.com>
Cc: Michal Marek <mmarek@suse.cz>
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

I installed Fedora 17 which no longer supports grub v1. I worked
with grub2 for a while, but there's so many issues with it and automated
rebooting, that I decided to switch to syslinux. Instead of using
the REBOOT_SCRIPT and add customized changes to get syslinux booted,
I thought it better to make ktest aware of syslinux and add options
to simplify the use of syslinux on a target test box.
Acked-by: NH. Peter Anvin <hpa@zytor.com>
Cc: John 'Warthog9' Hawley <warthog9@kernel.org>
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Before rebooting the target, run the sync command, as it seems that
either Grub2 or systemd gets screwed up if you update to reboot a kernel
once and do a reboot without doing a sync.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

As only grub or 'script' is supported for rebooting to a new kernel,
and Fedora 17 has dropped support for grub, I decided to add grub2
support as well (I also plan on adding syslinux/extlinux support too).

The options GRUB_FILE and GRUB_REBOOT were added to allow the user
to specify where to find the grub.cfg and what tool to use to reboot
into the next kernel respectively.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

In order to decide if ktest should bother installing modules on the
target box, it checks if the config file has CONFIG_MODULES=y. But it
also checks if the '=y' part exists. It only will install modules if the
config exists and is set with '=y'. But as the regex that was used
tests:

  /^CONFIG_MODULES(=y)?/

this will also match:

  CONFIG_MODULES_USE_ELF_RELA

as the '=y' part was optional and it did not test the rest of the line.
When this happens, ktest will stop checking the rest of the configs but
it will also think that no modules are needed to be installed. What it
should do is only jump out of the loop if it actually found a
CONFIG_MODULES that is set to true.

Otherwise, ktest wont install the necessary modules needed for proper
booting of the test target.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

As 67d34a6a said, 'oldnoconfig' doesn't
set new symbols to 'n', but instead sets it to their default values.

So, this patch replaces 'oldnoconfig' with 'olddefconfig', stop making
people confused, and keep the old name 'oldnoconfig' as an alias,
because people already are dependent on its behavior with the
counter-intuitive name.
Signed-off-by: NAdam Lee <adam8157@gmail.com>
Signed-off-by: NMichal Marek <mmarek@suse.cz>

The ELSE IF statements do not work as expected if another ELSE statement
follows. This is because the $if_set is not set. If the ELSE IF
condition is true, the following ELSE should be ignored. But because the
$if_set is not set, the following ELSE will also be executed.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Add '=~' and '!~' to the list of allowed conditionals for DEFAULT and
TEST_START section if statements.

ie.

 TEST_START IF TEST =~ .*test$
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

The option IGNORE_ERRORS is used to allow a test to succeed even if a
warning appears from the kernel. Sometimes kernels will produce warnings
that are not associated with a test, and the user wants to test
something else.

The IGNORE_ERRORS works for boot up, but was not preventing test runs to
succeed if the kernel produced a warning.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

The min configs are saved in a perl hash called force_configs, and this
hash is used to add configs to the .config file. But it was not being
reset between tests and a min config from a previous test would affect
the min config of the next test causing undesirable results.

Reset the force_config hash at the start of each test.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Usually the target is booted into a dependable kernel when a test
starts. The test will install the test kernel and reboot the box. But
there may be a time that the kernel is running an unreliable kernel and
the reboot may crash.

Have ktest detect crashes on a reboot and force a power-cycle instead.

This can usually happen if a test kernel was installed to run manual
tests, but the user forgot to reboot to the known good kernel.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

If the console is constantly outputting content, this can cause ktest
to get stuck waiting on the monitor to settle down.

The option MAX_MONITOR_WAIT is the maximum time (in seconds) for ktest
to wait for the console to flush.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

With a name like 'oldnoconfig' one may think that the config generated
would disable all configs that were not defined (selecting "no" for all
options). But this is not the case. It selects the default. If a config
has a 'default y', then it is added if not specified.

This broke the config bisect, because options not specified by a config
will just use the default, where it expected to turn off. This caused an
option to be enabled that disabled an option that would break the build.
The end result was that we never found the bad config at the end of the
test.

Instead of using 'make oldnoconfig', ktest now builds the options it
expects enabled and disabled. When it turns off an option, it will no
longer remove it, but actually set it to:

 # CONFIG_FOO is not set.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

The config-bisect can take a bad config and bisect it down to find out
what config actually breaks the config. But as all tests will apply a
minconfig (defined by a user) to apply before booting, it is possible
that the minconfig could actually make the bad config work (minconfigs
can disable configs). The end result is that the config bisect test will
not find a config that breaks. This can be rather frustrating to the
user.

The CONFIG_BISECT_CHECK option, when set to 1, will make sure that the
bad config (with the minconfig applied) still fails before trying to
bisect.

And yes, I did get burned by this.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Add the PRE_INSTALL option that will allow a user to specify a shell
command to be executed before the install operation executes.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

In order to let the user add commands before and after ktest runs, the
PRE_KTEST and POST_KTEST options are defined. They hold shell commands
that will execute befor ktest runs its first test, as well as when it
completed its last test.

The PRE_TEST and POST_TEST will be run befor and after (respectively)
for a given test. They can either be global (done for all tests) or
defined by a single test.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>