Context:
When using a non-UML arch, kunit.py will boot the test kernel with
options like these by default (this is x86_64):
> mem=1G console=tty kunit_shutdown=halt console=ttyS0 kunit_shutdown=reboot

The first three options are added unconditionally but are only intended
for UML.

1. 'mem=1G' is redundant with the '-m 1024' that we hard-code into the
   qemu commandline.

2. We specify a 'console' for all tools/testing/kunit/qemu_configs/*.py
   already, so 'console=tty' gets overwritten.

3. For QEMU, we need to use 'reboot', and for UML we need to use 'halt'.
   If you switch them, kunit.py will hang until the --timeout expires.

This patch:
Having these duplicate options is a bit noisy.
Switch so we only add UML-specific options for UML.

I.e. we now get
UML: 'mem=1G console=tty kunit_shutdown=halt' (unchanged)
x86_64: 'console=ttyS0 kunit_shutdown=reboot'

Side effect: you can't overwrite these options on UML w/ --kernel_arg.
But you already couldn't for QEMU (console, kunit_shutdown), and why
would you want to?
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Context:
* kunit_kernel.py is importing kunit_parser.py just to use the
  print_with_timestamp() function
* the parser is directly printing to stdout, which will become an issue
  if we ever try to run multiple kernels in parallel

This patch introduces a kunit_printer.py file and migrates callers of
kunit_parser.print_with_timestamp() to call
kunit_printer.stdout.print_with_timestamp() instead.

Future changes:
If we want to support showing results for parallel runs, we could then
create new Printer's that don't directly write to stdout and refactor
the code to pass around these Printer objects.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

It's always set to true except in one test case.
And in that test case it can safely be set to true anyways.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

This primarily comes from running pylint over kunit tool code and
ignoring some warnings we don't care about.
If we ever got a fully clean setup, we could add this to run_checks.py,
but we're not there yet.

Fix things like
* Drop unused imports
* check `is None`, not `== None` (see PEP 8)
* remove redundant parens around returns
* remove redundant `else` / convert `elif` to `if` where appropriate
* rename make_arch_qemuconfig() param to base_kunitconfig (this is the
  name used in the subclass, and it's a better one)
* kunit_tool_test: check the exit code for SystemExit (could be 0)
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Note: this potentially breaks custom qemu_configs if people are using
them! But the fix for them is simple, don't specify multiple arguments
in one string and don't add on a redundant ''.

It feels a bit iffy to be using a shell in the first place.

There's the usual shenanigans where people could pass in arbitrary shell
commands via --kernel_arg (since we're just adding '' around the
kernel_cmdline) or via a custom qemu_config.
This isn't too much of a concern given the nature of this script (and
the qemu_config file is in python, you can do w/e you want already).

But it does have some other drawbacks.

One example of a kunit-specific pain point:
If the relevant qemu binary is missing, we get output like this:
> /bin/sh: line 1: qemu-system-aarch64: command not found
This in turn results in our KTAP parser complaining about
missing/invalid KTAP, but we don't directly show the error!
It's even more annoying to debug when you consider --raw_output only
shows KUnit output by default, i.e. you need --raw_output=all to see it.

Whereas directly invoking the binary, Python will raise a
FileNotFoundError for us, which is a noisier but more clear.

Making this change requires
* splitting parameters like ['-m 256'] into ['-m', '256'] in
  kunit/qemu_configs/*.py
* change [''] to [] in kunit/qemu_configs/*.py since otherwise
  QEMU fails w/ 'Device needs media, but drive is empty'
* dropping explicit quoting of the kernel cmdline
* using shlex.quote() when we print what command we're running
  so the user can copy-paste and run it
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Before, kunit.py always printed "arch": "UM" in its json output, but...
1. With `kunit.py parse`, we could be parsing output from anywhere, so
    we can't say that.
2. Capitalizing it is probably wrong, as it's `ARCH=um`
3. Commit 87c9c163 ("kunit: tool: add support for QEMU") made it so
   kunit.py could knowingly run a different arch, yet we'd still always
   claim "UM".

This patch addresses all of those. E.g.

1.
$ ./tools/testing/kunit/kunit.py parse .kunit/test.log --json | grep -o '"arch.*' | sort -u
"arch": "",

2.
$ ./tools/testing/kunit/kunit.py run --json | ...
"arch": "um",

3.
$ ./tools/testing/kunit/kunit.py run --json --arch=x86_64 | ...
"arch": "x86_64",
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

--build_dir is set to a default of '.kunit' since commit ddbd60c7
("kunit: use --build_dir=.kunit as default"), but even before then it
was explicitly set to ''.

So outside of one unit test, there was no way for the build_dir to be
ever be None, and we can simplify code by fixing the unit test and
enforcing that via updated type annotations.

E.g. this lets us drop `get_file_path()` since it's now exactly
equivalent to os.path.join().

Note: there's some `if build_dir` checks that also fail if build_dir is
explicitly set to '' that just guard against passing "O=" to make.
But running `make O=` works just fine, so drop these checks.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Python 3.10.0 contains:
9e09849d20 ("bpo-41006: importlib.util no longer imports typing (GH-20938)")

It causes importlib.util to no longer import importlib.abs, which leads
to the following error when trying to use kunit with qemu:
AttributeError: module 'importlib' has no attribute 'abc'. Did you mean: '_abc'?

Add the missing import.
Signed-off-by: NMichał Winiarski <michal.winiarski@intel.com>
Reviewed-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

After upgrading mypy and pytype from pip, we see 2 new errors when
running ./tools/testing/kunit/run_checks.py.

Error #1: mypy and pytype
They now deduce that importlib.util.spec_from_file_location() can return
None and note that we're not checking for this.

We validate that the arch is valid (i.e. the file exists) beforehand.
Add in an `asssert spec is not None` to appease the checkers.

Error #2: pytype bug https://github.com/google/pytype/issues/1057
It doesn't like `from datetime import datetime`, specifically that a
type shares a name with a module.

We can workaround this by either
* renaming the import or just using `import datetime`
* passing the new `--fix-module-collisions` flag to pytype.

We pick the first option for now because
* the flag is quite new, only in the 2021.11.29 release.
* I'd prefer if people can just run `pytype <file>`
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Problem: currently, if you remove something from your kunitconfig,
kunit.py will not regenerate the .config file.
The same thing happens if you did --kunitconfig_add=CONFIG_KASAN=y [1]
and then ran again without it. Your new run will still have KASAN.

The reason is that kunit.py won't regenerate the .config file if it's a
superset of the kunitconfig. This speeds it up a bit for iterating.

This patch adds an additional check that forces kunit.py to regenerate
the .config file if the current kunitconfig doesn't match the previous
one.

What this means:
* deleting entries from .kunitconfig works as one would expect
* dropping  a --kunitconfig_add also triggers a rebuild
* you can still edit .config directly to turn on new options

We implement this by creating a `last_used_kunitconfig` file in the
build directory (so .kunit, by default) after we generate the .config.
When comparing the kconfigs, we compare python sets, so duplicates and
permutations don't trip us up.

The majority of this patch is adding unit tests for the existing logic
and for the new case where `last_used_kunitconfig` differs.

[1] https://lore.kernel.org/linux-kselftest/20211106013058.2621799-2-dlatypov@google.com/Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

The current error message is precise, but not very clear if you don't
already know what it's talking about, e.g.

> $ make ARCH=um olddefconfig O=.kunit
> ERROR:root:Provided Kconfig is not contained in validated .config. Following fields found in kunitconfig, but not in .config: CONFIG_DRM=y

Try to reword the error message so that it's
* your missing options usually have unsatisified dependencies
* if you're on UML, that might be the cause (it is, in this example)
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

E.g. run tests but with KASAN
$ ./tools/testing/kunit/kunit.py run --arch=x86_64 --kconfig_add=CONFIG_KASAN=y

This also works with --kunitconfig
$ ./tools/testing/kunit/kunit.py run --arch=x86_64 --kunitconfig=fs/ext4 --kconfig_add=CONFIG_KASAN=y

This flag is inspired by TuxMake's --kconfig-add, see
https://gitlab.com/Linaro/tuxmake#examples.

Our version just uses "_" as the delimiter for consistency with
pre-existing flags like --build_dir, --make_options, --kernel_args, etc.

Note: this does make it easier to run into a pre-existing edge case:
$ ./tools/testing/kunit/kunit.py run --arch=x86_64 --kconfig_add=CONFIG_KASAN=y
$ ./tools/testing/kunit/kunit.py run --arch=x86_64
This second invocation ^ still has KASAN enabled!

kunit.py won't call olddefconfig if our current .config is already a
superset of the provided kunitconfig.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

read_from_file() clears its `self` Kconfig object and parses a config
file.

It is a way to construct Kconfig objects more so than an operation on
Kconfig objects. This is reflected in the fact its only ever used as:
  kconfig = kunit_config.Kconfig()
  kconfig.read_from_file(path)

So clean this up and simplify callers by replacing it with
  kconfig = kunit_config.parse_file(path)

Do the same thing for the related parse_from_string() function as well.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Currently, we have these errors:
$ mypy ./tools/testing/kunit/*.py
tools/testing/kunit/kunit_kernel.py:213: error: Item "_Loader" of "Optional[_Loader]" has no attribute "exec_module"
tools/testing/kunit/kunit_kernel.py:213: error: Item "None" of "Optional[_Loader]" has no attribute "exec_module"
tools/testing/kunit/kunit_kernel.py:214: error: Module has no attribute "QEMU_ARCH"
tools/testing/kunit/kunit_kernel.py:215: error: Module has no attribute "QEMU_ARCH"

exec_module
===========

pytype currently reports no errors, but that's because there's a comment
disabling it on 213.

This is due to https://github.com/python/typeshed/pull/2626.
The fix is to assert the loaded module implements the ABC
(abstract base class) we want which has exec_module support.

QEMU_ARCH
=========

pytype is fine with this, but mypy is not:
https://github.com/python/mypy/issues/5059

Add a check that the loaded module does indeed have QEMU_ARCH.
Note: this is not enough to appease mypy, so we also add a comment to
squash the warning.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

kunit.py currently crashes and fails to parse kernel output if it's not
fully valid utf-8.

This can come from memory corruption or just inadvertently printing
out binary data as strings.

E.g. adding this line into a kunit test
  pr_info("\x80")
will cause this exception
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position
  1961: invalid start byte

We can tell Python how to handle errors, see
https://docs.python.org/3/library/codecs.html#error-handlers

Unfortunately, it doesn't seem like there's a way to specify this in
just one location, so we need to repeat ourselves quite a bit.

Specify `errors='backslashreplace'` so we instead:
* print out the offending byte as '\x80'
* try and continue parsing the output.
  * as long as the TAP lines themselves are valid, we're fine.

Fixed spelling/grammar in commit log:
Shuah Khan <<skhan@linuxfoundation.org>
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Tested-by: NDavid Gow <davidgow@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Currently, `run_kernel()` dumps all the kernel output to a file
(.kunit/test.log) and then opens the file and yields it to callers.
This made it easier to respect the requested timeout, if any.

But it means that we can't yield the results in real time, either to the
parser or to stdout (if --raw_output is set).

This change spins up a background thread to enforce the timeout, which
allows us to yield the kernel output in real time, while also copying it
to the .kunit/test.log file.
It's also careful to ensure that the .kunit/test.log file is complete,
even in the kunit_parser throws an exception/otherwise doesn't consume
every line, see the new `finally` block and unit test.

For example:

$ ./tools/testing/kunit/kunit.py run --arch=x86_64 --raw_output
<configure + build steps>
...
<can now see output from QEMU in real time>

This does not currently have a visible effect when --raw_output is not
passed, as kunit_parser.py currently only outputs everything at the end.
But that could change, and this patch is a necessary step towards
showing parsed test results in real time.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Consider this attempt to run KUnit in QEMU:
$ ./tools/testing/kunit/kunit.py run --arch=x86

Before you'd get this error message:
kunit_kernel.ConfigError: x86 is not a valid arch

After:
kunit_kernel.ConfigError: x86 is not a valid arch, options are ['alpha', 'arm', 'arm64', 'i386', 'powerpc', 'riscv', 's390', 'sparc', 'x86_64']

This should make it a bit easier for people to notice when they make
typos, etc. Currently, one would have to dive into the python code to
figure out what the valid set is.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Drop some variables in unit tests that were unused and/or add assertions
based on them.

For ExitStack, it was imported, but the `es` variable wasn't used so it
didn't do anything, and we were leaking the file objects.
Refactor it to just use nested `with` statements to properly close them.

And drop the direct use of .close() on file objects in the kunit tool
unit test, as these can be leaked if test assertions fail.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

The import was working around the fact "tuple[T]" was used instead of
typing.Tuple[T].

Convert it to use type.Tuple to be consistent with how the rest of the
code is anotated.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

The default .kunitconfig file is currently kept in
arch/um/configs/kunit_defconfig, but -- with the impending QEMU patch
-- will no-longer be exclusively used for UML-based kernels.

Move it alongside the other KUnit configs in
tools/testing/kunit/configs, and give it a name which matches the
existing all_tests.config and broken_on_uml.config files.

Also update the Getting Started documentation to point to the new file.
Signed-off-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Add basic support to run QEMU via kunit_tool. Add support for i386,
x86_64, arm, arm64, and a bunch more.
Signed-off-by: NBrendan Higgins <brendanhiggins@google.com>
Tested-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Add a new kernel command-line option, 'kunit_shutdown', which allows the
user to specify that the kernel poweroff, halt, or reboot after
completing all KUnit tests; this is very handy for running KUnit tests
on UML or a VM so that the UML/VM process exits cleanly immediately
after running all tests without needing a special initramfs.
Signed-off-by: NDavid Gow <davidgow@google.com>
Signed-off-by: NBrendan Higgins <brendanhiggins@google.com>
Reviewed-by: NStephen Boyd <sboyd@kernel.org>
Tested-By: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDaniel Latypov <dlatypov@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

TL;DR
$ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit

Per suggestion from Ted [1], we can reduce the amount of typing by
assuming a convention that these files are named '.kunitconfig'.

In the case of [1], we now have
$ ./tools/testing/kunit/kunit.py run --kunitconfig=fs/ext4

Also add in such a fragment for kunit itself so we can give that as an
example more close to home (and thus less likely to be accidentally
broken).

[1] https://lore.kernel.org/linux-ext4/YCNF4yP1dB97zzwD@mit.edu/Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

This is a bug that has been present since the first version of this
code.
Using [] as a default parameter is dangerous, since it's mutable.

Example using the REPL:
>>> def bad(param = []):
...     param.append(len(param))
...     print(param)
...
>>> bad()
[0]
>>> bad()
[0, 1]

This wasn't a concern in the past since it would just keep appending the
same values to it.

E.g. before, `args` would just grow in size like:
  [mem=1G', 'console=tty']
  [mem=1G', 'console=tty', mem=1G', 'console=tty']

But with now filter_glob, this is more dangerous, e.g.
  run_kernel(filter_glob='my-test*') # default modified here
  run_kernel()			     # filter_glob still applies here!
That earlier `filter_glob` will affect all subsequent calls that don't
specify `args`.

Note: currently the kunit tool only calls run_kernel() at most once, so
it's not possible to trigger any negative side-effects right now.

Fixes: 6ebf5866 ("kunit: tool: add Python wrappers for running KUnit tests")
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

This allows running different subsets of tests, e.g.

$ ./tools/testing/kunit/kunit.py build
$ ./tools/testing/kunit/kunit.py exec 'list*'
$ ./tools/testing/kunit/kunit.py exec 'kunit*'

This passes the "kunit_filter.glob" commandline option to the UML
kernel, which currently only supports filtering by suite name.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Currently running tests via KUnit tool means tweaking a .kunitconfig
file, which you'd keep around locally and never commit.
This changes makes it so users can pass in a path to a kunitconfig.

One of the imagined use cases is having kunitconfig fragments in-tree
to formalize interesting sets of tests for features/subsystems, e.g.
  $ ./tools/testing/kunit/kunit.py run --kunticonfig=fs/ext4/kunitconfig

For now, this hypothetical fs/ext4/kunitconfig would contain
  CONFIG_KUNIT=y
  CONFIG_EXT4_FS=y
  CONFIG_EXT4_KUNIT_TESTS=y

At the moment, it's not hard to manually whip up this file, but as more
and more tests get added, this will get tedious.

It also opens the door to documenting how to run all the tests relevant
to a specific subsystem or feature as a simple one-liner.

This can be seen as an analogue to tools/testing/selftests/*/config
But in the case of KUnit, the tests live in the same directory as the
code-under-test, so it feels more natural to allow the kunitconfig
fragments to live anywhere. (Though, people could create a separate
directory if wanted; this patch imposes no restrictions on the path).
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

LinuxSourceTree will unceremoniously crash if the user doesn't call
read_kunitconfig() first in a number of functions.

And currently every place we create an instance, the caller also calls
create_kunitconfig() and read_kunitconfig().
Move these instead into __init__() so they can't be forgotten and to
reduce copy-paste.

The https://github.com/google/pytype type-checker complained that
_config wasn't initialized. With this, kunit_tool now type checks
under both pytype and mypy.

Add an optional boolean that can be used to disable this for use cases
in the future where we might not need/want to load the config.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

The authors of this tool were more familiar with a different
type-checker, https://github.com/google/pytype.

That's open source, but mypy seems more prevalent (and runs faster).
And unlike pytype, mypy doesn't try to infer types so it doesn't check
unanotated functions.

So annotate ~all functions in kunit tool to increase type-checking
coverage.
Note: per https://www.python.org/dev/peps/pep-0484/, `__init__()` should
be annotated as `-> None`.

Doing so makes mypy discover a number of new violations.
Exclude main() since we reuse `request` for the different types of
requests, which mypy isn't happy about.

This commit fixes all but one error, where `TestSuite.status` might be
None.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Acked-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

kunit_tool relies on the UML console outputting printk() output to the
tty in order to get results. Since the default console driver could
change, pass 'console=tty' to the kernel.

This is triggered by a change[1] to use ttynull as a fallback console
driver which -- by chance or by design -- seems to have changed the
default console output on UML, breaking kunit_tool. While this may be
fixed, we should be less fragile to such changes in the default.

[1]:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=757055ae8dedf5333af17b3b5b4b70ba9bc9da4eSigned-off-by: NDavid Gow <davidgow@google.com>
Fixes: 757055ae ("init/console: Use ttynull as a fallback when there is no console")
Reported-by: NAndy Shevchenko <andriy.shevchenko@linux.intel.com>
Tested-by: NAndy Shevchenko <andriy.shevchenko@linux.intel.com>
Acked-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Helper allows to derive file names depending on --build_dir argument.
Signed-off-by: NAndy Shevchenko <andriy.shevchenko@linux.intel.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Currently the tool redirects make stdout + stderr, and only shows them
if the make command fails.
This means build warnings aren't shown to the user.

This change prints the contents of stderr even if make succeeds, under
the assumption these are only build warnings or other messages the user
likely wants to see.

We drop stdout from the raised exception since we can no longer easily
collate stdout and stderr and just showing the stderr seems fine.

Example with a warning:

[14:56:35] Building KUnit Kernel ...
../lib/kunit/kunit-test.c: In function ‘kunit_test_successful_try’:
../lib/kunit/kunit-test.c:19:6: warning: unused variable ‘unused’ [-Wunused-variable]
   19 |  int unused;
      |      ^~~~~~

[14:56:40] Starting KUnit Kernel ...

Note the stderr has a trailing \n, and since we use print, we add
another, but it helps separate make and kunit.py output.

Example with a build error:

[15:02:45] Building KUnit Kernel ...
ERROR:root:../lib/kunit/kunit-test.c: In function ‘kunit_test_successful_try’:
../lib/kunit/kunit-test.c:19:2: error: unknown type name ‘invalid_type’
   19 |  invalid_type *test = data;
      |  ^~~~~~~~~~~~
...
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NDavid Gow <davidgow@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

When --build_dir is provided use it and do not pollute source directory
which even can be mounted over network or read-only.
Signed-off-by: NAndy Shevchenko <andriy.shevchenko@linux.intel.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

When --build_dir is provided use it and do not pollute source directory
which even can be mounted over network or read-only.
Signed-off-by: NAndy Shevchenko <andriy.shevchenko@linux.intel.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

CalledProcessError stores the output of the failed process as `bytes`,
not a `str`.

So when we log it on build error, the make output is all crammed into
one line with "\n" instead of actually printing new lines.

After this change, we get readable output with new lines, e.g.
>   CC      lib/kunit/kunit-example-test.o
> In file included from ../lib/kunit/test.c:9:
> ../include/kunit/test.h:22:1: error: unknown type name ‘invalid_type_that_causes_compile’
>    22 | invalid_type_that_causes_compile errors;
>       | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> make[3]: *** [../scripts/Makefile.build:283: lib/kunit/test.o] Error 1

Secondly, trying to concat exceptions to strings will fail with
> TypeError: can only concatenate str (not "OSError") to str
so fix this with an explicit cast to str.
Signed-off-by: NDaniel Latypov <dlatypov@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Alltests flag evidently stopped working when run from outside of the
root of the source tree, so fix that. Also add an additional broken
config to the broken_on_uml config.
Signed-off-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Direct stderr to subprocess.STDOUT so error messages get included in the
subprocess.CalledProcessError exceptions output field. This results in
more meaningful error messages for the user.

This is already being done in the make_allyesconfig method. Do the same
for make_mrproper, make_olddefconfig, and make methods.

With this, failures on unclean trees [1] will give users an error
message that includes:
"The source tree is not clean, please run 'make ARCH=um mrproper'"

[1] https://bugzilla.kernel.org/show_bug.cgi?id=205219Signed-off-by: NWill Chen <chenwi@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

The kunit.py utility builds an ARCH=um kernel and then runs it.  Add
optional --make_options flag to kunit.py allowing for the operator to
specify extra build options.

This allows use of the clang compiler for kunit:
  tools/testing/kunit/kunit.py run --defconfig \
    --make_options CC=clang --make_options HOSTCC=clang
Signed-off-by: NGreg Thelen <gthelen@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Reviewed-by: NNathan Chancellor <natechancellor@gmail.com>
Tested-by: NDavid Gow <davidgow@google.com>
Signed-off-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Implemented the functionality to run all KUnit tests through kunit_tool
by specifying an --alltests flag, which builds UML with allyesconfig
enabled, and consequently runs every KUnit test. A new function was
added to kunit_kernel: make_allyesconfig.
Firstly, if --alltests is specified, kunit.py triggers build_um_kernel
which call make_allyesconfig. This function calls the make command,
disables the broken configs that would otherwise prevent UML from
building, then starts the kernel with all possible configurations
enabled. All stdout and stderr is sent to test.log and read from there
then fed through kunit_parser to parse the tests to the user. Also added
a signal_handler in case kunit is interrupted while running.
Tested: Run under different conditions such as testing with
--raw_output, testing program interrupt then immediately running kunit
again without --alltests and making sure to clean the console.
Signed-off-by: NHeidi Fahim <heidifahim@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

Previous error message for invalid kunitconfig was vague. Added to it so
that it lists invalid fields and prompts for them to be removed.  Added
validate_config function returning whether or not this kconfig is valid.
Signed-off-by: NHeidi Fahim <heidifahim@google.com>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>

This commit renames 'kunitconfig' to '.kunitconfig' so that it can be
automatically ignored by git and do not disturb people who want to type
'kernel/' by pressing only the 'k' and then 'tab' key.
Signed-off-by: NSeongJae Park <sjpark@amazon.de>
Reviewed-by: NBrendan Higgins <brendanhiggins@google.com>
Tested-by: NBrendan Higgins <brendanhiggins@google.com>
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>