Currently setting do_not_reboot is triggered by simple builds and bisect
builds, but not config bisect builds.

Link: http://lkml.kernel.org/r/20170717001630.10518-3-swood@redhat.comSigned-off-by: NScott Wood <swood@redhat.com>
Signed-off-by: NSteven Rostedt (VMware) <rostedt@goodmis.org>

Rather than adding a third copy of the same logic, rework it to cover
all three buildonly cases at once.

In the future, please consider using the same variable to perform the
same function regardless of context...

Link: http://lkml.kernel.org/r/20170717001630.10518-2-swood@redhat.comSigned-off-by: NScott Wood <swood@redhat.com>
Signed-off-by: NSteven Rostedt (VMware) <rostedt@goodmis.org>

Simply telling a new user to edit "the config file" without giving any
hints on where that file should go, what it should be named, or where
a template can be found, is not particularly helpful.

Link: http://lkml.kernel.org/r/20170717001630.10518-1-swood@redhat.comSigned-off-by: NScott Wood <swood@redhat.com>
Signed-off-by: NSteven Rostedt (VMware) <rostedt@goodmis.org>

Before ktest issues a reboot, it will try to connect to the target machine
to make sure that it is still alive. If the target does not respond within 5
seconds, it will power cycle the box instead of issuing a reboot.

Five seconds may be too short, and ktest may unnecessarially power cycle the
box. I have found 25 seconds seems to be a better timeout for this purpose.
But even 25 may be too arbitrary. Add a CONNECT_TIMEOUT option to let the
user determine the timeout time before rebooting. By default, it has been
raised to 25 seconds.
Signed-off-by: NSteven Rostedt (VMware) <rostedt@goodmis.org>

To clean up the console processes that are forked to monitor the console,
there needs to be a waitpid().
Signed-off-by: NSteven Rostedt (VMware) <rostedt@goodmis.org>

The function wait_for_input takes in a timeout, and even has a default
timeout. But if for some reason the STDIN descriptor keeps sending in data,
the function will never time out. The timout is to wait for the data from
the passed in file descriptor, not for STDIN. Adding a test in the case
where there's no data from the passed in file descriptor that checks to see
if the timeout passed, will ensure that it will timeout properly even if
there's input in STDIN.
Signed-off-by: NSteven Rostedt (VMware) <rostedt@goodmis.org>

The run_command function was changed to use the wait_for_input function to
allow having a timeout if the command to run takes too much time. There was
a bug in the wait_for_input where it could end up going into an infinite
loop. There's two issues here. One is that the return value of the sysread
wasn't used for the write (to write a proper size), and that it should
continue processing the passed in file descriptor too even if there was
input. There was no check for error, if for some reason STDIN returned an
error, the function would go into an infinite loop and never exit.
Reported-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>
Tested-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>
Fixes: 6e98d1b4 ("ktest: Add timeout to ssh command")
Signed-off-by: NSteven Rostedt (VMware) <rostedt@goodmis.org>

Create a variable called run_command_status that saves the status of the
executed commands and can be used by other functions later to test for
status.
Signed-off-by: NSteven Rostedt (VMware) <rostedt@goodmis.org>

When performing a reboot of the test box, try to ssh to it. If it can't
connect for 5 seconds, then powercycle the box. This is useful because the
reboot is done via ssh, and if you can't ssh to the box because it is hung,
the reboot fails to reboot.
Signed-off-by: NSteven Rostedt (VMware) <rostedt@goodmis.org>

Add a timeout to performing an ssh command. This will let testing if a
machine is alive or not, or if something else may be amiss. A timeout can be
passed to ssh, where ssh will fail if it does not complete within the given
timeout.
Signed-off-by: NSteven Rostedt (VMware) <rostedt@goodmis.org>

The child_exit errno needs to be shifted by 8 bits to compare against the
return values for the bisect variables.

Fixes: c5dacb88 ("ktest: Allow overriding bisect test results")
Cc: stable@vger.kernel.org
Signed-off-by: NSteven Rostedt (VMware) <rostedt@goodmis.org>

The POST_TEST config is to be executed after a test has fully compeleted,
whether the test passed or failed. It currently is executed at the moment
that the test has been decided if it failed or not. As the test does other
clean ups, it isn't truly finished. Move the POST_TEST execution to after
all the test cleanups have been done.
Signed-off-by: NSteven Rostedt (VMware) <rostedt@goodmis.org>

Ajdust spelling to more common "mandatory".  Variant "mandidory" is
certainly wrong.

Link: http://lkml.kernel.org/r/20161011073003.GA19476@amdSigned-off-by: NPavel Machek <pavel@ucw.cz>
Acked-by: NSteven Rostedt <rostedt@goodmis.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Seems that some of the new console logic causes doprint to possibly
get evaluated. When printing a commit message that contains parenthesis,
it fails with a shell parsing error.

This gets fixed when we add quotes around the $item variable, and prevent
it from being evaluated by any shell commands.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

If dodie() is called with the console open, restore the terminal's
original settings before dying.

Link: http://lkml.kernel.org/r/20150130025453.GB20952@treble.redhat.comSigned-off-by: NJosh Poimboeuf <jpoimboe@redhat.com>
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Since both success and failure may shortcut and exit ktest, it is better
to print the status times there too. Once times are printed, the values
for the times are reset, so they will not print more than once.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Allow the user to send input to the console by putting the terminal in
cbreak mode (to allow reading stdin one character at a time) and copying
all stdin data to the console's pty.

Link: http://lkml.kernel.org/r/bb1bbe7d202c95a3ce7894cfffdd8c725875978e.1422473610.git.jpoimboe@redhat.comSigned-off-by: NJosh Poimboeuf <jpoimboe@redhat.com>
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Create a pseudoterminal (pty pair) to give the console a dedicated tty
so it doesn't mess with ktest's terminal settings.

Link: http://lkml.kernel.org/r/37b0127f9efad09ff4fc994334db998141e4f6ca.1422473610.git.jpoimboe@redhat.comSigned-off-by: NJosh Poimboeuf <jpoimboe@redhat.com>
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

The function start_monitor_and_boot is a misnomer. It use to, but
now it starts the monitor and installs. It does not boot. Rename it
before I get confused by it again.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Seeing the times for how long a build, install, reboot and the
test takes is helpful for analyzing the test process. Seeing
how different changes affect the timings.

Show the build, install, boot and test times when at the end of
the test, or between each interval for tests that do those
mulitple times (like bisect and patchcheck).
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

When ktest runs the console program as a child process, the parent and
child share the same tty for stdin and stderr.  This is problematic when
using a libvirt target.  The "virsh console" program makes a lot of
changes to the tty settings, making ktest's output hard to read
(carriage returns don't work).  After ktest exits, the terminal is
unusable (CRs broken, stdin isn't echoed).

I think the best way to fix this issue would be to create a
pseudoterminal (pty pair) so the child process would have a dedicated
tty, and then use pipes to connect the two ttys.  I'm not sure if that's
overkill, but it's far beyond my current Perl abilities.

This patch is a much easier way to (partially) fix this issue.  It saves
the tty settings before opening the console and restores them after
closing it.  There are still a few places where ktest prints mangled
output while the console is open, but the output is much more legible
overall, and the terminal works just fine after ktest exits.

Link: http://lkml.kernel.org/r/1bb89abc0025cf1d6da657c7ba58bbeb4381a515.1422382008.git.jpoimboe@redhat.comSigned-off-by: NJosh Poimboeuf <jpoimboe@redhat.com>
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

I find that I usually like to see how long a make or other command takes,
and adding a start and end time and reporting how long each command runs
(in seconds) is helpful.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Commit 52d21580 "ktest: Use make -s kernelrelease" fixed commit
7ff52571 "kbuild: fake the "Entering directory ..." message more simply"
as that commit added output after the make kernelrelease. But there's still
some build scripts that are used by ktest that has output before the make
is executed, and requires that only the last line is printed.

Cc: Michal Marek <mmarek@suse.cz>
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Instead of just showing the test type of test in the start of the
test, like this:

  RUNNING TEST 1 of 26 with option build defconfig

Add the name (if it is defined) as well, like this:

  RUNNING TEST 1 of 26 (arm64 aarch64-linux) with option build defconfig
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Tests can set options that override the default ones. But if a test
tries to undefine a default option, it is simply ignored and the
default option stays as is.

For example, if you want to have a test that defines no MIN_CONFIG
then the test should be able to do that with:

   TEST_START
   MIN_CONFIG =

Which should make MIN_CONFIG not defined for that test. But the way
the code currently works, undefined options in tests are dropped.
This is because the NULL options are evaluated during the reading of
the config file and since one can disable default options in the default
section with this method, it is evaluated there (the option turns to a
undef). But undef options in the test section mean to use the default
option.

To fix this, keep the empty string in the option during the reading
of the config file, and then evaluate it when running the test. This
will allow tests to null out default options.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Commit 6071c22e "ktest: Rewrite the config-bisect to actually work"
fixed the config-bisect to work nicely but in doing so it broke
make_min_config by changing the way assign_configs works.

The assign_configs function now adds the config to the hash even if
it is disabled, but changes the hash value to be that of the
line "# CONFIG_FOO is not set". Unfortunately, the make_min_config
test only checks to see if the config is removed. It now needs to
check if the config is in the hash and not set to be disabled.

Cc: stable@vger.kernel.org # 3.17+
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

The previous tail -1 broke with commit 7ff52571 ("kbuild: fake the
"Entering directory ..." message more simply")

Link: http://lkml.kernel.org/r/20141022194408.GA20989@pobox.suse.czReported-by: NSteven Rostedt <rostedt@goodmis.org>
Signed-off-by: NMichal Marek <mmarek@suse.cz>
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

If git bisect reply is being used in the bisect tests, don't bother
doing the git bisect good or git bisect bad calls. The git bisect
reply will override them anyway, and that's called immediately
after the other two. Going the git bisect (good|bad) is just a
waste of time.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

The reboot function when rebooting back to a good kernel has a check
to make sure that a new kernel was indeed booted. But that check
uses a timeout value, which when calling the monitor will still
return success if the timeout is hit (no bug was found). It should
return an error to let the reboot code know that a new kernel was
not reached. Only the reboot code checks the return value of the
monitor.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

When doing a manual bisect, a build can fail or a test can be inconclusive.
In these cases it would be helpful to be able to skip the test entirely.

Link: http://lkml.kernel.org/r/1409164021-2136-1-git-send-email-chris.j.arges@canonical.comReviewed-by: NSatoru Takeuchi <satoru.takeuchi@gmail.com>
Signed-off-by: NChris J Arges <chris.j.arges@canonical.com>
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Add a way to run a patchcheck test on the commits that are in one branch
but not in another. This uses git cherry to find a list of commits to
test each one with.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

After the rewrite of the config bisect, the bisect manual was
removed. Add it back.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

After the rewrite of the config bisect, there were several unused
functions that can be removed.

One of the unused functions printed out the failed config nicer than
what the rewrite did, so I kept that and used it to output the
bad config.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

The new rewrite left out the CONFIG_BISECT_CHECK, which allows the
user to test that their "bad" config still is bad and their "good"
config still is good. This is especially important as the configs
are passed through a "make oldconfig" to update them with the lastest
kernel. Things could change that causes a bad config to work, or a
good config to break. The check is done after the configs have run
through the oldconfig processing.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

I never liked the way config-bisect worked. I would assume the bad config
had some config that broke the system. But it would not work if the bad
config just happened to be missing something that the good config had.

I rewrote the config-bisect to do this properly. It does a diff of the two
configs, and sets half of the configs that are in one and not the other.
The way it works is that when it "sets", it really just makes one copy
what the other has. That is, a "set" can be setting a:

  # CONFIG_FOO is not set

Basically, it looks at the differences between the two files and makes
them similar until it comes down to one config that makes it work or
not work depending on if it is set or not.

Note, if more than one config change makes the bad config not work, it
will only find one of them. But this is true with all bisect logic.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Some cleanup for improving readability as follows.

  - Initialize $ktest_config at its definition.
  - Put parentheses around the `config-file' argument in the usage message
    because it's a optional one.
  - Rename get_ktest_config{,s} to more descriptive get_mandatory_config{,s}.

Link: http://lkml.kernel.org/r/87fvmr30kb.wl%satoru.takeuchi@gmail.comSigned-off-by: NSatoru Takeuchi <satoru.takeuchi@gmail.com>
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

If we'd like to set the redirect target file of run_command(),
we should define $redirect before calling this function and should undef it
after calling this function. Since it's user-unfriendly, add 2nd parameter of
run_command() for this purpose.

Link: http://lkml.kernel.org/r/87vbvwokq8.wl%satoru.takeuchi@gmail.comSigned-off-by: NSatoru Takeuchi <satoru.takeuchi@gmail.com>
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

For those cases that it takes several tries to hit a bug, it would be
useful for ktest.pl to try a test multiple times before it considers
the test as a pass. To accomplish this, BISECT_TRIES ktest config
option has been added. It is default to one, as most of the time a
bisect only needs to try a test once. But the user can now up this
to make ktest run a given test multiple times. The first failure
that is detected will set a bisect bad. It only repeats on success.

Note, as with all race bugs, there's no guarantee that if it succeeds,
it is really a good bisect. But it helps in case the bug is somewhat
reliable.

You can set BISECT_TRIES to zero, and all tests will be considered
good, unless you also set BISECT_MANUAL.
Suggested-by: N"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

With the added variable ${KERNEL_VERSION}, it is useful to be
able to use parts of it for other variables.

For example, if you want to create a warnings file for each major
kernel version to test sub versions against you can create
your warnings file with like this:

  WARNINGS_FILE = warnings-file-${KERNEL_VERSION}

But this may add 3.8.12 or something, and we want all 3.8.* to
use the same file, and 3.10.* to use another file, and so on.
With the eval command we can, by adding:

  WARNINGS_FILE =~ s/(-file-\d+\.\d+).*/$1/

Which will chop off the extra characters after the 3.8.
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>

Add a special variable that can be used in other variables called
${KERNEL_VERSION}. This will embed the current kernel version into
the variable. For example:

WARNINGS_FILE = ${OUTPUT_DIR}/warnings-${KERNEL_VERSION}

If the current version is v3.8 then the WARNINGS_FILE will become

  ${OUTPUT_DIR}/warnings-v3.8
Signed-off-by: NSteven Rostedt <rostedt@goodmis.org>