Problem: currently, if you remove something from your kunitconfig,
kunit.py will not regenerate the .config file.
The same thing happens if you did --kunitconfig_add=CONFIG_KASAN=y [1]
and then ran again without it. Your new run will still have KASAN.

The reason is that kunit.py won't regenerate the .config file if it's a
superset of the kunitconfig. This speeds it up a bit for iterating.

This patch adds an additional check that forces kunit.py to regenerate
the .config file if the current kunitconfig doesn't match the previous
one.

What this means:
* deleting entries from .kunitconfig works as one would expect
* dropping  a --kunitconfig_add also triggers a rebuild
* you can still edit .config directly to turn on new options

We implement this by creating a `last_used_kunitconfig` file in the
build directory (so .kunit, by default) after we generate the .config.
When comparing the kconfigs, we compare python sets, so duplicates and
permutations don't trip us up.

The majority of this patch is adding unit tests for the existing logic
and for the new case where `last_used_kunitconfig` differs.

[1] https://lore.kernel.org/linux-kselftest/20211106013058.2621799-2-dlatypov@google.com/Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

E.g. run tests but with KASAN
$ ./tools/testing/kunit/kunit.py run --arch=x86_64 --kconfig_add=CONFIG_KASAN=y

This also works with --kunitconfig
$ ./tools/testing/kunit/kunit.py run --arch=x86_64 --kunitconfig=fs/ext4 --kconfig_add=CONFIG_KASAN=y

This flag is inspired by TuxMake's --kconfig-add, see
https://gitlab.com/Linaro/tuxmake#examples.

Our version just uses "_" as the delimiter for consistency with
pre-existing flags like --build_dir, --make_options, --kernel_args, etc.

Note: this does make it easier to run into a pre-existing edge case:
$ ./tools/testing/kunit/kunit.py run --arch=x86_64 --kconfig_add=CONFIG_KASAN=y
$ ./tools/testing/kunit/kunit.py run --arch=x86_64
This second invocation ^ still has KASAN enabled!

kunit.py won't call olddefconfig if our current .config is already a
superset of the provided kunitconfig.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

read_from_file() clears its `self` Kconfig object and parses a config
file.

It is a way to construct Kconfig objects more so than an operation on
Kconfig objects. This is reflected in the fact its only ever used as:
  kconfig = kunit_config.Kconfig()
  kconfig.read_from_file(path)

So clean this up and simplify callers by replacing it with
  kconfig = kunit_config.parse_file(path)

Do the same thing for the related parse_from_string() function as well.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

With the parser rework [1] and run_kernel() rework [2], this allows the
parser to print out test results incrementally.

Currently, that's held up by the fact that the LineStream eagerly
pre-fetches the next line when you call pop().
This blocks parse_test_result() from returning until the line *after*
the "ok 1 - test name" line is also printed.

One can see this with the following example:
$ (echo -e 'TAP version 14\n1..3\nok 1 - fake test'; sleep 2; echo -e 'ok 2 - fake test 2'; sleep 3; echo -e 'ok 3 - fake test 3') | ./tools/testing/kunit/kunit.py parse

Before this patch [1]: there's a pause before 'fake test' is printed.
After this patch: 'fake test' is printed out immediately.

This patch also adds
* a unit test to verify LineStream's behavior directly
* a test case to ensure that it's lazily calling the generator
* an explicit exception for when users go beyond EOF

[1] https://lore.kernel.org/linux-kselftest/20211006170049.106852-1-dlatypov@google.com/
[2] https://lore.kernel.org/linux-kselftest/20211005011340.2826268-1-dlatypov@google.com/Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

It's possible for a test to have a subtest header, but zero valid
subtests. We used to error on this if the test plan had no subtests
listed, but it's possible to have subtests without a test plan (indeed,
this is how parameterised tests work).

Tests with 0 subtests now have the result NO_TESTS, and will report an
error (which does not halt test execution, but is printed in a scary red
colour and is noted in the results summary).
Signed-off-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

The (K)TAP spec encourages test output to begin with a 'test plan': a
count of the number of tests being run of the form:
1..n

However, some test suites might not know the number of subtests in
advance (for example, KUnit's parameterised tests use a generator
function). In this case, it's not possible to print the test plan in
advance.

kunit_tool already parses test output which doesn't contain a plan, but
reports an error. Since we want to use nested subtests with KUnit
paramterised tests, remove this error.
Signed-off-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Currently, KUnit will report SKIPPED tests as having failed if one uses
--json.

Add the missing if statement to set the appropriate status ("SKIP").
See https://api.kernelci.org/schema-test-case.html:
  "status": {
      "type": "string",
      "description": "The status of the execution of this test case",
      "enum": ["PASS", "FAIL", "SKIP", "ERROR"],
      "default": "PASS"
  },
with this, we now can properly produce all four of the statuses.

Fixes: 5acaf603 ("kunit: tool: Support skipped tests in kunit_tool")
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Update to kunit_parser to improve compatibility with KTAP
specification including arbitrarily nested tests. Patch accomplishes
three major changes:

- Use a general Test object to represent all tests rather than TestCase
and TestSuite objects. This allows for easier implementation of arbitrary
levels of nested tests and promotes the idea that both test suites and test
cases are tests.

- Print errors incrementally rather than all at once after the
parsing finishes to maximize information given to the user in the
case of the parser given invalid input and to increase the helpfulness
of the timestamps given during printing. Note that kunit.py parse does
not print incrementally yet. However, this fix brings us closer to
this feature.

- Increase compatibility for different formats of input. Arbitrary levels
of nested tests supported. Also, test cases and test suites are now
supported to be present on the same level of testing.

This patch now implements the draft KTAP specification here:
https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJxqaJk+r-K1YJzPggFDQ@mail.gmail.com/
We'll update the parser as the spec evolves.

This patch adjusts the kunit_tool_test.py file to check for
the correct outputs from the new parser and adds a new test to check
the parsing for a KTAP result log with correct format for multiple nested
subtests (test_is_test_passed-all_passed_nested.log).

This patch also alters the kunit_json.py file to allow for arbitrarily
nested tests.
Signed-off-by: NRae Moar <rmoar@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Currently, `run_kernel()` dumps all the kernel output to a file
(.kunit/test.log) and then opens the file and yields it to callers.
This made it easier to respect the requested timeout, if any.

But it means that we can't yield the results in real time, either to the
parser or to stdout (if --raw_output is set).

This change spins up a background thread to enforce the timeout, which
allows us to yield the kernel output in real time, while also copying it
to the .kunit/test.log file.
It's also careful to ensure that the .kunit/test.log file is complete,
even in the kunit_parser throws an exception/otherwise doesn't consume
every line, see the new `finally` block and unit test.

For example:

$ ./tools/testing/kunit/kunit.py run --arch=x86_64 --raw_output
<configure + build steps>
...
<can now see output from QEMU in real time>

This does not currently have a visible effect when --raw_output is not
passed, as kunit_parser.py currently only outputs everything at the end.
But that could change, and this patch is a necessary step towards
showing parsed test results in real time.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

The new --run_isolated flag makes the tool boot the kernel once per
suite or test, preventing leftover state from one suite to impact the
other. This can be useful as a starting point to debugging test
hermeticity issues.

Note: it takes a lot longer, so people should not use it normally.

Consider the following very simplified example:

  bool disable_something_for_test = false;
  void function_being_tested() {
    ...
    if (disable_something_for_test) return;
    ...
  }

  static void test_before(struct kunit *test)
  {
    disable_something_for_test = true;
    function_being_tested();
    /* oops, we forgot to reset it back to false */
  }

  static void test_after(struct kunit *test)
  {
    /* oops, now "fixing" test_before can cause test_after to fail! */
    function_being_tested();
  }

Presented like this, the issues are obvious, but it gets a lot more
complicated to track down as the amount of test setup and helper
functions increases.

Another use case is memory corruption. It might not be surfaced as a
failure/crash in the test case or suite that caused it. I've noticed in
kunit's own unit tests, the 3rd suite after might be the one to finally
crash after an out-of-bounds write, for example.

Example usage:

Per suite:
$ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit --run_isolated=suite
...
Starting KUnit Kernel (1/7)...
============================================================
======== [PASSED] kunit_executor_test ========
....
Testing complete. 5 tests run. 0 failed. 0 crashed. 0 skipped.
Starting KUnit Kernel (2/7)...
============================================================
======== [PASSED] kunit-try-catch-test ========
...

Per test:
$ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit --run_isolated=test
Starting KUnit Kernel (1/23)...
============================================================
======== [PASSED] kunit_executor_test ========
[PASSED] parse_filter_test
============================================================
Testing complete. 1 tests run. 0 failed. 0 crashed. 0 skipped.
Starting KUnit Kernel (2/23)...
============================================================
======== [PASSED] kunit_executor_test ========
[PASSED] filter_subsuite_test
...

It works with filters as well:
$ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit --run_isolated=suite example
...
Starting KUnit Kernel (1/1)...
============================================================
======== [PASSED] example ========
...

It also handles test filters, '*.*skip*' runs these 3 tests:
  kunit_status.kunit_status_mark_skipped_test
  example.example_skip_test
  example.example_mark_skipped_test

Fixed up merge conflict between:
  d8c23ead ("kunit: tool: better handling of quasi-bool args (--json, --raw_output)") and
  6710951ee039 ("kunit: tool: support running each suite/test separately")
Reported-by: NStephen Rothwell <sfr@canb.auug.org.au>
    Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Consider this attempt to run KUnit in QEMU:
$ ./tools/testing/kunit/kunit.py run --arch=x86

Before you'd get this error message:
kunit_kernel.ConfigError: x86 is not a valid arch

After:
kunit_kernel.ConfigError: x86 is not a valid arch, options are ['alpha', 'arm', 'arm64', 'i386', 'powerpc', 'riscv', 's390', 'sparc', 'x86_64']

This should make it a bit easier for people to notice when they make
typos, etc. Currently, one would have to dive into the python code to
figure out what the valid set is.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Drop some variables in unit tests that were unused and/or add assertions
based on them.

For ExitStack, it was imported, but the `es` variable wasn't used so it
didn't do anything, and we were leaking the file objects.
Refactor it to just use nested `with` statements to properly close them.

And drop the direct use of .close() on file objects in the kunit tool
unit test, as these can be leaked if test assertions fail.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Problem:

What does this do?
$ kunit.py run --json
Well, it runs all the tests and prints test results out as JSON.

And next is
$ kunit.py run my-test-suite --json
This runs just `my-test-suite` and prints results out as JSON.

But what about?
$ kunit.py run --json my-test-suite
This runs all the tests and stores the json results in a "my-test-suite"
file.

Why:
--json, and now --raw_output are actually string flags. They just have a
default value. --json in particular takes the name of an output file.

It was intended that you'd do
$ kunit.py run --json=my_output_file my-test-suite
if you ever wanted to specify the value.

Workaround:
It doesn't seem like there's a way to make
https://docs.python.org/3/library/argparse.html only accept arg values
after a '='.

I believe that `--json` should "just work" regardless of where it is.
So this patch automatically rewrites a bare `--json` to `--json=stdout`.

That makes the examples above work the same way.
Add a regression test that can catch this for --raw_output.

Fixes: 6a499c9c ("kunit: tool: make --raw_output support only showing kunit output")
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Tested-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

--raw_output is nice, but it would be nicer if could show only output
after KUnit tests have started.

So change the flag to allow specifying a string ('kunit').
Make it so `--raw_output` alone will default to `--raw_output=all` and
have the same original behavior.

Drop the small kunit_parser.raw_output() function since it feels wrong
to put it in "kunit_parser.py" when the point of it is to not parse
anything.

E.g.

$ ./tools/testing/kunit/kunit.py run --raw_output=kunit
...
[15:24:07] Starting KUnit Kernel ...
TAP version 14
1..1
    # Subtest: example
    1..3
    # example_simple_test: initializing
    ok 1 - example_simple_test
    # example_skip_test: initializing
    # example_skip_test: You should not see a line below.
    ok 2 - example_skip_test # SKIP this test should be skipped
    # example_mark_skipped_test: initializing
    # example_mark_skipped_test: You should see a line below.
    # example_mark_skipped_test: You should see this line.
    ok 3 - example_mark_skipped_test # SKIP this test should be skipped
ok 1 - example
[15:24:10] Elapsed time: 6.487s total, 0.001s configuring, 3.510s building, 0.000s running
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

kunit.py currently does not make it possible for users to specify module
parameters (/kernel arguments more generally) unless one directly tweaks
the kunit.py code itself.

This hasn't mattered much so far, but this would make it easier to port
existing tests that expose module parameters over to KUnit and/or let
current KUnit tests take advantage of them.

Tested using an kunit internal parameter:
$ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit \
    --kernel_args=kunit.filter_glob=kunit_status
...
Testing complete. 2 tests run. 0 failed. 0 crashed. 0 skipped.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

This patch addresses misleading error messages reported by kunit_tool in
two cases. First, in the case of TAP output having an incorrect header
format or missing a header, the parser used to output an error message of
'no tests run!'. Now the parser outputs an error message of 'could not
parse test results!'.

As an example:

Before:
$ ./tools/testing/kunit/kunit.py parse /dev/null
[ERROR] no tests run!
...

After:
$ ./tools/testing/kunit/kunit.py parse /dev/null
[ERROR] could not parse test results!
...

Second, in the case of TAP output with the correct header but no
tests, the parser used to output an error message of 'could not parse
test results!'. Now the parser outputs an error message of 'no tests
run!'.

As an example:

Before:
$ echo -e 'TAP version 14\n1..0' | ./tools/testing/kunit/kunit.py parse
[ERROR] could not parse test results!

After:
$ echo -e 'TAP version 14\n1..0' | ./tools/testing/kunit/kunit.py parse
[ERROR] no tests run!

Additionally, this patch also corrects the tests in kunit_tool_test.py
and adds a test to check the error in the case of TAP output with the
correct header but no tests.
Signed-off-by: NRae Moar <rmoar@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Add support for the SKIP directive to kunit_tool's TAP parser.

Skipped tests now show up as such in the printed summary. The number of
skipped tests is counted, and if all tests in a suite are skipped, the
suite is also marked as skipped. Otherwise, skipped tests do affect the
suite result.

Example output:
[00:22:34] ======== [SKIPPED] example_skip ========
[00:22:34] [SKIPPED] example_skip_test # SKIP this test should be skipped
[00:22:34] [SKIPPED] example_mark_skipped_test # SKIP this test should be skipped
[00:22:34] ============================================================
[00:22:34] Testing complete. 2 tests run. 0 failed. 0 crashed. 2 skipped.
Signed-off-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Note: this does not change the parser behavior at all (except for making
one error message more useful). This is just an internal refactor.

The TAP output parser currently operates over a List[str].
This works, but we only ever need to be able to "peek" at the current
line and the ability to "pop" it off.

Also, using a List means we need to wait for all the output before we
can start parsing. While this is not an issue for most tests which are
really lightweight, we do have some longer (~5 minutes) tests.

This patch introduces an LineStream wrapper class that
* Exposes a peek()/pop() interface instead of manipulating an array
  * this allows us to more easily add debugging code [1]
* Can consume an input from a generator
  * we can now parse results as tests are running (the parser code
  currently doesn't print until the end, so no impact yet).
* Tracks the current line number to print better error messages
* Would allow us to add additional features more easily, e.g. storing
  N previous lines so we can print out invalid lines in context, etc.

[1] The parsing logic is currently quite fragile.
E.g. it'll often say the kernel "CRASHED" if there's something slightly
wrong with the output format. When debugging a test that had some memory
corruption issues, it resulted in very misleading errors from the parser.

Now we could easily add this to trace all the lines consumed and why
+import inspect
...
        def pop(self) -> str:
                n = self._next
+               print(f'popping {n[0]}: {n[1].ljust(40, " ")}| caller={inspect.stack()[1].function}')

Example output:
popping 77: TAP version 14                          | caller=parse_tap_header
popping 78: 1..1                                    | caller=parse_test_plan
popping 79:     # Subtest: kunit_executor_test      | caller=parse_subtest_header
popping 80:     1..2                                | caller=parse_subtest_plan
popping 81:     ok 1 - parse_filter_test            | caller=parse_ok_not_ok_test_case
popping 82:     ok 2 - filter_subsuite_test         | caller=parse_ok_not_ok_test_case
popping 83: ok 1 - kunit_executor_test              | caller=parse_ok_not_ok_test_suite

If we introduce an invalid line, we can see the parser go down the wrong path:
popping 77: TAP version 14                          | caller=parse_tap_header
popping 78: 1..1                                    | caller=parse_test_plan
popping 79:     # Subtest: kunit_executor_test      | caller=parse_subtest_header
popping 80:     1..2                                | caller=parse_subtest_plan
popping 81:     1..2 # this is invalid!             | caller=parse_ok_not_ok_test_case
popping 82:     ok 1 - parse_filter_test            | caller=parse_ok_not_ok_test_case
popping 83:     ok 2 - filter_subsuite_test         | caller=parse_ok_not_ok_test_case
popping 84: ok 1 - kunit_executor_test              | caller=parse_ok_not_ok_test_case
[ERROR] ran out of lines before end token
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Acked-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Add basic support to run QEMU via kunit_tool. Add support for i386,
x86_64, arm, arm64, and a bunch more.
Signed-off-by: NBrendan Higgins <brendanhiggins@google.com>
Tested-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

TL;DR
$ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit

Per suggestion from Ted [1], we can reduce the amount of typing by
assuming a convention that these files are named '.kunitconfig'.

In the case of [1], we now have
$ ./tools/testing/kunit/kunit.py run --kunitconfig=fs/ext4

Also add in such a fragment for kunit itself so we can give that as an
example more close to home (and thus less likely to be accidentally
broken).

[1] https://lore.kernel.org/linux-ext4/YCNF4yP1dB97zzwD@mit.edu/Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

This allows running different subsets of tests, e.g.

$ ./tools/testing/kunit/kunit.py build
$ ./tools/testing/kunit/kunit.py exec 'list*'
$ ./tools/testing/kunit/kunit.py exec 'kunit*'

This passes the "kunit_filter.glob" commandline option to the UML
kernel, which currently only supports filtering by suite name.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Currently running tests via KUnit tool means tweaking a .kunitconfig
file, which you'd keep around locally and never commit.
This changes makes it so users can pass in a path to a kunitconfig.

One of the imagined use cases is having kunitconfig fragments in-tree
to formalize interesting sets of tests for features/subsystems, e.g.
  $ ./tools/testing/kunit/kunit.py run --kunticonfig=fs/ext4/kunitconfig

For now, this hypothetical fs/ext4/kunitconfig would contain
  CONFIG_KUNIT=y
  CONFIG_EXT4_FS=y
  CONFIG_EXT4_KUNIT_TESTS=y

At the moment, it's not hard to manually whip up this file, but as more
and more tests get added, this will get tedious.

It also opens the door to documenting how to run all the tests relevant
to a specific subsystem or feature as a simple one-liner.

This can be seen as an analogue to tools/testing/selftests/*/config
But in the case of KUnit, the tests live in the same directory as the
code-under-test, so it feels more natural to allow the kunitconfig
fragments to live anywhere. (Though, people could create a separate
directory if wanted; this patch imposes no restrictions on the path).
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Also take this time to rename get_absolute_path() to test_data_path().

1. the name is currently a lie. It gives relative paths, e.g. if I run
from the same dir as the test file, it gives './test_data/<file>'

See https://docs.python.org/3/reference/import.html#__file__, which
doesn't stipulate that implementations provide absolute paths.

2. it's only used for generating paths to tools/testing/kunit/test_data/
So we can tersen things by making it less general.

Cache the absolute path to the test data files per suggestion from  [1].
Using relative paths, the tests break because of this code in kunit.py
  if get_kernel_root_path():
          os.chdir(get_kernel_root_path())

[1] https://lore.kernel.org/linux-kselftest/CABVgOSnH0gz7z5JhRCGyG1wg0zDDBTLoSUCoB-gWMeXLgVTo2w@mail.gmail.com/

Fixes: 5578d008 ("kunit: tool: fix running kunit_tool from outside kernel tree")
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Acked-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

The use of manual open() and .close() calls seems to be an attempt to
keep the contents in scope.
But Python doesn't restrict variables like that, so we can introduce new
variables inside of a `with` and use them outside.

Do so to make the code more Pythonic.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Acked-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Use self.assertEqual/assertNotEqual() instead.
Besides being more appropriate in a unit test, it'll also give a better
error message by show the unexpected values.

Also
* Delete redundant check of exception types. self.assertRaises does this.
* s/kall/call. There's no reason to name it this way.
  * This is probably a misunderstanding from the docs which uses it
  since `mock.call` is in scope as `call`.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Acked-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

* Stop leaking file objects.
* Use self.addCleanup() to ensure we call cleanup functions even if
setUp() fails.
* use mock.patch.stopall instead of more error-prone manual approach
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Acked-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Change every shebang which does not need an argument to use /usr/bin/env.
This is needed as not every distro has everything under /usr/bin,
sometimes not even bash.
Signed-off-by: NFinn Behrens <me@kloenk.de>
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

For simplcity, strip all trailing whitespace from parsed output.
I imagine no one is printing out meaningful trailing whitespace via
KUNIT_FAIL() or similar, and that if they are, they really shouldn't.

`isolate_kunit_output()` yielded liens with trailing \n, which results
in artifacty output like this:

$ ./tools/testing/kunit/kunit.py run
[16:16:46] [FAILED] example_simple_test
[16:16:46]     # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29

[16:16:46]     Expected 1 + 1 == 3, but

[16:16:46]         1 + 1 == 2

[16:16:46]         3 == 3

[16:16:46]     not ok 1 - example_simple_test

[16:16:46]

After this change:
[16:16:46]     # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
[16:16:46]     Expected 1 + 1 == 3, but
[16:16:46]         1 + 1 == 2
[16:16:46]         3 == 3
[16:16:46]     not ok 1 - example_simple_test
[16:16:46]

We should *not* be expecting lines to end with \n in kunit_tool_test.py
for this reason.

Do the same for `raw_output()` as well which suffers from the same
issue.

This is a followup to [1], but rebased onto kunit-fixes to pick up the
other raw_output() fix and fixes for kunit_tool_test.py.

[1] https://lore.kernel.org/linux-kselftest/20201020233219.4146059-1-dlatypov@google.com/Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Tested-by: NDavid Gow <davidgow@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Some tests logs for kunit_tool tests are missing their test plans
causing their tests to fail; fix this by adding the test plans.

Fixes: 45dcbb6f ("kunit: test: add test plan to KUnit TAP format")
Signed-off-by: NBrendan Higgins <brendanhiggins@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Add a --json flag, which when specified generates JSON formatted test
results conforming to the KernelCI API test_group spec[1]. The user can
use the new flag to specify a filename to print the json formatted
results to.

Link[1]: https://api.kernelci.org/schema-test-group.html#postSigned-off-by: NHeidi Fahim <heidifahim@google.com>
Signed-off-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Commit 01397e82 ("kunit: Fix TabError, remove defconfig code and
handle when there is no kunitconfig") and commit 45ba7a89 ("kunit:
kunit_tool: Separate out config/build/exec/parse") introduced two
closely related issues which built off of each other: they excessively
created the build directory when not present and modified a constant
(constants in Python only exist by convention).

Together these issues broken a number of unit tests for KUnit tool, so
fix them.

Fixed up commit log to fic checkpatch commit description style error.
Shuah Khan <skhan@linuxfoundation.org>

Fixes: 01397e82 ("kunit: Fix TabError, remove defconfig code and handle when there is no kunitconfig")
Fixes: 45ba7a89 ("kunit: kunit_tool: Separate out config/build/exec/parse")
Signed-off-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Commit ddbd60c7 ("kunit: use --build_dir=.kunit as default") changed
the default build directory for KUnit tests, but failed to update
associated unit tests for kunit_tool, so update them.

Fixes: ddbd60c7 ("kunit: use --build_dir=.kunit as default")
Signed-off-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Currently, if the kernel is configured incorrectly or if it crashes before any
kunit tests are run, kunit finishes without error, reporting
that 0 test cases were run.

To fix this, an error is shown when the tap header is not found, which
indicates that kunit was not able to run at all.
Signed-off-by: NUriel Guajardo <urielguajardo@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Add new subcommands to kunit.py to allow stages of the existing 'run'
subcommand to be run independently:
- 'config': Verifies that .config is a subset of .kunitconfig
- 'build': Compiles a UML kernel for KUnit
- 'exec': Runs the kernel, and outputs the test results.
- 'parse': Parses test results from a file or stdin

The 'run' command continues to behave as before.
Signed-off-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Rework kunit_tool in order to allow .kunitconfig files to better enforce
that disabled items in .kunitconfig are disabled in the generated
.config.

Previously, kunit_tool simply enforced that any line present in
.kunitconfig was also present in .config, but this could cause problems
if a config option was disabled in .kunitconfig, but not listed in .config
due to (for example) having disabled dependencies.

To fix this, re-work the parser to track config names and values, and
require values to match unless they are explicitly disabled with the
"CONFIG_x is not set" comment (or by setting its value to 'n'). Those
"disabled" values will pass validation if omitted from the .config, but
not if they have a different value.
Signed-off-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Implemented the functionality to run all KUnit tests through kunit_tool
by specifying an --alltests flag, which builds UML with allyesconfig
enabled, and consequently runs every KUnit test. A new function was
added to kunit_kernel: make_allyesconfig.
Firstly, if --alltests is specified, kunit.py triggers build_um_kernel
which call make_allyesconfig. This function calls the make command,
disables the broken configs that would otherwise prevent UML from
building, then starts the kernel with all possible configurations
enabled. All stdout and stderr is sent to test.log and read from there
then fed through kunit_parser to parse the tests to the user. Also added
a signal_handler in case kunit is interrupted while running.
Tested: Run under different conditions such as testing with
--raw_output, testing program interrupt then immediately running kunit
again without --alltests and making sure to clean the console.
Signed-off-by: NHeidi Fahim <heidifahim@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Previously, kunit_parser did not properly handle kunit TAP output that
- had any prefixes (generated from different configs e.g.
CONFIG_PRINTK_TIME)
- had unrelated kernel output mixed in the middle of
it, which has shown up when testing with allyesconfig
To remove prefixes, the parser looks for the first line that includes
TAP output, "TAP version 14".  It then determines the length of the
string before this sequence, and strips that number of characters off
the beginning of the following lines until the last KUnit output line is
reached.
These fixes have been tested with additional tests in the
KUnitParseTest and their associated logs have also been added.
Signed-off-by: NHeidi Fahim <heidifahim@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

This commit adds kunit tool test for the '--build_dir' option.
Signed-off-by: NSeongJae Park <sjpark@amazon.de>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

'kunit' writes the 'test.log' under the kernel source directory even
though a 'build_dir' option is given.  As users who use the option might
expect the outputs to be placed under the specified directory, this
commit modifies the logic to write the log file under the 'build_dir'.
Signed-off-by: NSeongJae Park <sjpark@amazon.de>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Assert in test_run_timeout was not updated with the build_dir argument
and caused the following error:
AssertionError: Expected call: run_kernel(timeout=3453)
Actual call: run_kernel(build_dir=None, timeout=3453)

Needed to update kunit_tool_test to reflect this fix
https://lkml.org/lkml/2019/9/6/351Signed-off-by: NHeidi Fahim <heidifahim@google.com>
Reviewed-by: NSeongJae Park <sjpark@amazon.de>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>