Rename overlay-y to multi-dtb-y, which is a consistent name with
multi-obj-y. Also, use multi-search to avoid code duplication.

Introduce real-dtb-y, which is a consistent name with real-obj-y,
to contain primitive blobs compiled from *.dts. This is used to
calculate the list of *.dt.yaml files.

Set -@ to base DTB without using $(eval ).
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

Make include/config/foo/bar.h fake deps files generation simpler.

* delete .h suffix
	those aren't header files, shorten filenames,

* delete tolower()
	Linux filesystems can deal with both upper and lowercase
	filenames very well,

* put everything in 1 directory
	Presumably 'mkdir -p' split is from dark times when filesystems
	handled huge directories badly, disks were round adding to
	seek times.

	x86_64 allmodconfig lists 12364 files in include/config.

	../obj/include/config/
	├── 104_QUAD_8
	├── 60XX_WDT
	├── 64BIT
		...
	├── ZSWAP_DEFAULT_ON
	├── ZSWAP_ZPOOL_DEFAULT
	└── ZSWAP_ZPOOL_DEFAULT_ZBUD

	0 directories, 12364 files
Signed-off-by: NAlexey Dobriyan <adobriyan@gmail.com>
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

I think multi-obj-* is clearer, and more consistent with real-obj-*.

Rename as follows:

  multi-used-y  ->  multi-obj-y
  multi-used-m  ->  multi-obj-m
  multi-used    ->  multi-obj-ym
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

scripts/Kbuild.include is included everywhere, but macros such as
cc-option are needed by build targets only.

For example, when 'make clean' traverses the tree, it does not need
to evaluate $(call cc-option,).

Split cc-option, ld-option, etc. to scripts/Makefile.compiler, which
is only included from the top Makefile and scripts/Makefile.build.
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

VPATH is used in Kbuild to make pattern rules search for prerequisites
in both $(objtree) and $(srctree). Some of *.c, *.S files are not real
sources, but generated by tools such as flex, bison, perl.

In contrast, I doubt the benefit of --include-dir=$(abs_srctree) because
it is always clear which Makefiles are real sources, and which are not.

So, my hope is to add $(srctree)/ prefix to all check-in Makefiles,
then remove --include-dir=$(abs_srctree) flag in the future.

I am touching only some Kbuild core parts for now. Treewide fixes will
be needed to achieve this goal.
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

These have no more user in the upstream code. The use of them has been
warned for a while for external modules. The migration is finished.
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

With LTO, LLVM bitcode won't be compiled into native code until
modpost_link, or modfinal for modules. This change postpones calls
to objtool until after these steps, and moves objtool_args to
Makefile.lib, so the arguments can be reused in Makefile.modfinal.

As we didn't have objects to process earlier, we use --duplicate
when processing vmlinux.o. This change also disables unreachable
instruction warnings with LTO to avoid warnings about the int3
padding between functions.
Signed-off-by: NSami Tolvanen <samitolvanen@google.com>
Reviewed-by: NKees Cook <keescook@chromium.org>

This change adds build support for using objtool to generate
__mcount_loc sections.
Signed-off-by: NSami Tolvanen <samitolvanen@google.com>

With CONFIG_MODVERSIONS, version information is linked into each
compilation unit that exports symbols. With LTO, we cannot use this
method as all C code is compiled into LLVM bitcode instead. This
change collects symbol versions into .symversions files and merges
them in link-vmlinux.sh where they are all linked into vmlinux.o at
the same time.
Signed-off-by: NSami Tolvanen <samitolvanen@google.com>
Reviewed-by: NKees Cook <keescook@chromium.org>
Signed-off-by: NKees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20201211184633.3213045-4-samitolvanen@google.com

This change adds build system support for Clang's Link Time
Optimization (LTO). With -flto, instead of ELF object files, Clang
produces LLVM bitcode, which is compiled into native code at link
time, allowing the final binary to be optimized globally. For more
details, see:

  https://llvm.org/docs/LinkTimeOptimization.html

The Kconfig option CONFIG_LTO_CLANG is implemented as a choice,
which defaults to LTO being disabled. To use LTO, the architecture
must select ARCH_SUPPORTS_LTO_CLANG and support:

  - compiling with Clang,
  - compiling all assembly code with Clang's integrated assembler,
  - and linking with LLD.

While using CONFIG_LTO_CLANG_FULL results in the best runtime
performance, the compilation is not scalable in time or
memory. CONFIG_LTO_CLANG_THIN enables ThinLTO, which allows
parallel optimization and faster incremental builds. ThinLTO is
used by default if the architecture also selects
ARCH_SUPPORTS_LTO_CLANG_THIN:

  https://clang.llvm.org/docs/ThinLTO.html

To enable LTO, LLVM tools must be used to handle bitcode files, by
passing LLVM=1 and LLVM_IAS=1 options to make:

  $ make LLVM=1 LLVM_IAS=1 defconfig
  $ scripts/config -e LTO_CLANG_THIN
  $ make LLVM=1 LLVM_IAS=1

To prepare for LTO support with other compilers, common parts are
gated behind the CONFIG_LTO option, and LTO can be disabled for
specific files by filtering out CC_FLAGS_LTO.
Signed-off-by: NSami Tolvanen <samitolvanen@google.com>
Reviewed-by: NKees Cook <keescook@chromium.org>
Signed-off-by: NKees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20201211184633.3213045-3-samitolvanen@google.com

Move function tracer options to Kconfig to make it easier to add
new methods for generating __mcount_loc, and to make the options
available also when building kernel modules.

Note that FTRACE_MCOUNT_USE_* options are updated on rebuild and
therefore, work even if the .config was generated in a different
environment.
Signed-off-by: NSami Tolvanen <samitolvanen@google.com>
Acked-by: NSteven Rostedt (VMware) <rostedt@goodmis.org>
Signed-off-by: NKees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20201211184633.3213045-2-samitolvanen@google.com

"xargs echo" is not a safe way to remove line breaks because the input
may exceed the command line limit and xargs may break it up into
multiple invocations of echo. This should never happen because
scripts/gen_autoksyms.sh expects all undefined symbols are placed in
the second line of .mod files.

One possible way is to replace "xargs echo" with
"sed ':x;N;$!bx;s/\n/ /g'" or something, but I rewrote the code by
using awk because it is more readable.

This issue was reported by Sami Tolvanen; in his Clang LTO patch set,
$(multi-used-m) is no longer an ELF object, but a thin archive that
contains LLVM bitcode files. llvm-nm prints out symbols for each
archive member separately, which results a lot of dupications, in some
places, beyond the system-defined limit.

This problem must be fixed irrespective of LTO, and we must ensure
zero possibility of having this issue.

Link: https://lkml.org/lkml/2020/12/1/1658Reported-by: NSami Tolvanen <samitolvanen@google.com>
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>
Reviewed-by: NSami Tolvanen <samitolvanen@google.com>

This change removes all instances of DISABLE_LTO from
Makefiles, as they are currently unused, and the preferred
method of disabling LTO is to filter out the flags instead.

Note added by Masahiro Yamada:
DISABLE_LTO was added as preparation for GCC LTO, but GCC LTO was
not pulled into the mainline. (https://lkml.org/lkml/2014/4/8/272)
Suggested-by: NKees Cook <keescook@chromium.org>
Signed-off-by: NSami Tolvanen <samitolvanen@google.com>
Reviewed-by: NKees Cook <keescook@chromium.org>
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

The conditional:

  ifneq ($(hostprogs),)

... is evaluated to true if $(hostprogs) does not contain any word but
whitespace characters.

  ifneq ($(strip $(hostprogs)),)

... is a safe way to avoid interpreting whitespace as a non-empty value,
but I'd rather want to use the side-effect of $(sort ...) to do the
equivalent.

$(sort ...) is used in scripts/Makefile.host in order to drop duplication
in $(hostprogs). It is also useful to strip excessive spaces.

Move $(sort ...) before evaluating the ifneq.
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

The host shared library rules are currently implemented in
scripts/Makefile.host, but actually GCC-plugin is the only user of
them. (The VDSO .so files are built for the target by different
build rules) Hence, they do not need to be treewide available.

Move all the relevant build rules to scripts/gcc-plugins/Makefile.

I also optimized the build steps so *.so is directly built from .c
because every upstream plugin is compiled from a single source file.

I am still keeping the multi-file plugin support, which Kees Cook
mentioned might be needed by out-of-tree plugins.
(https://lkml.org/lkml/2019/1/11/1107)

If the plugin, foo.so, is compiled from two files foo.c and foo2.c,
then you can do like follows:

  foo-objs := foo.o foo2.o

Single-file plugins do not need the *-objs notation.
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>
Acked-by: NKees Cook <keescook@chromium.org>

Currently, the directories of objects are automatically created
only for O= builds.

It should not hurt to cater to this for in-tree builds too.
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

Since the pre-git time the checker is run first, before the compiler.
But if the source file contains some syntax error, the warnings from
the compiler are more useful than those from sparse (and other
checker most probably too).

So move the 'check' command to run after the compiler.
Signed-off-by: NLuc Van Oostenryck <luc.vanoostenryck@gmail.com>
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

Make modules.order depend on $(obj-m), and use if_changed to build it.
This will avoid unneeded update of modules.order, which will be useful
to optimize the modpost stage.

Currently, the second pass of modpost is always invoked. By checking the
timestamp of modules.order, we can avoid the unneeded modpost.
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

Remove lib-target, builtin-target, modorder-target, and modtargets.

Instead, add targets-for-builtin and targets-for-modules.
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

built-in.a contains the built-in object paths from the current and sub
directories.

module.order collects the module paths from the current and sub
directories.

Make their build rules look more symmetrical.
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

I think subdir-builtin is clearer.

While I was here, I made its build rule explicit.
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

Save $(addprefix ...) for subdir-obj-y.
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

Do not try to build any module-related artifacts when CONFIG_MODULES
is disabled.
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

I do not see a good reason to add ifdef here.
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

Kbuild supports the infrastructure to build host programs, but there
was no support to build userspace programs for the target architecture
(i.e. the same architecture as the kernel).

Sam Ravnborg worked on this in 2014 (https://lkml.org/lkml/2014/7/13/154),
but it was not merged. One problem at that time was, there was no good way
to know whether $(CC) can link standalone programs. In fact, pre-built
kernel.org toolchains [1] are often used for building the kernel, but they
do not provide libc.

Now, we can handle this cleanly because the compiler capability is
evaluated at the Kconfig time. If $(CC) cannot link standalone programs,
the relevant options are hidden by 'depends on CC_CAN_LINK'.

The implementation just mimics scripts/Makefile.host

The userspace programs are compiled with the same flags as the host
programs. In addition, it uses -m32 or -m64 if it is found in
$(KBUILD_CFLAGS).

This new syntax has two usecases.

- Sample programs

  Several userspace programs under samples/ include UAPI headers
  installed in usr/include. Most of them were previously built for
  the host architecture just to use the 'hostprogs' syntax.

  However, 'make headers' always works for the target architecture.
  This caused the arch mismatch in cross-compiling. To fix this
  distortion, sample code should be built for the target architecture.

- Bpfilter

  net/bpfilter/Makefile compiles bpfilter_umh as the user mode helper,
  and embeds it into the kernel. Currently, it overrides HOSTCC with
  CC to use the 'hostprogs' syntax. This hack should go away.

[1]: https://mirrors.edge.kernel.org/pub/tools/crosstool/Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>
Acked-by: NSam Ravnborg <sam@ravnborg.org>

Kbuild supports not only obj-y but also lib-y to list objects linked to
vmlinux.

The difference between them is that all the objects from obj-y are
forcibly linked to vmlinux, whereas the objects from lib-y are linked
as needed; if there is no user of a lib-y object, it is not linked.

lib-y is intended to list utility functions that may be called from all
over the place (and may be unused at all), but it is a problem for
EXPORT_SYMBOL(). Even if there is no call-site in the vmlinux, we need
to keep exported symbols for the use from loadable modules.

Commit 7f2084fa ("[kbuild] handle exports in lib-y objects reliably")
worked around it by linking a dummy object, lib-ksyms.o, which contains
references to all the symbols exported from lib.a in that directory.
It uses the linker script command, EXTERN. Unfortunately, the meaning of
EXTERN of ld.lld is different from that of ld.bfd. Therefore, this does
not work with LD=ld.lld (CBL issue #515).

Anyway, the build rule of lib-ksyms.o is somewhat tricky. So, I want to
get rid of it.

At first, I was thinking of accumulating lib-y objects into obj-y
(or even replacing lib-y with obj-y entirely), but the lib-y syntax
is used beyond the ordinary use in lib/ and arch/*/lib/.

Examples:

 - drivers/firmware/efi/libstub/Makefile builds lib.a, which is linked
   into vmlinux in the own way (arm64), or linked to the decompressor
   (arm, x86).

 - arch/alpha/lib/Makefile builds lib.a which is linked not only to
   vmlinux, but also to bootloaders in arch/alpha/boot/Makefile.

 - arch/xtensa/boot/lib/Makefile builds lib.a for use from
   arch/xtensa/boot/boot-redboot/Makefile.

One more thing, adding everything to obj-y would increase the vmlinux
size of allnoconfig (or tinyconfig).

For less impact, I tweaked the destination of lib.a at the top Makefile;
when CONFIG_MODULES=y, lib.a goes to KBUILD_VMLINUX_OBJS, which is
forcibly linked to vmlinux, otherwise lib.a goes to KBUILD_VMLINUX_LIBS
as before.

The size impact for normal usecases is quite small since at lease one
symbol in every lib-y object is eventually called by someone. In case
you are intrested, here are the figures.

x86_64_defconfig:

   text	   data	    bss	    dec	    hex	filename
19566602 5422072 1589328 26578002 1958c52 vmlinux.before
19566932 5422104 1589328 26578364 1958dbc vmlinux.after

The case with the biggest impact is allnoconfig + CONFIG_MODULES=y.

ARCH=x86 allnoconfig + CONFIG_MODULES=y:

   text	   data	    bss	    dec	    hex	filename
1175162	 254740	1220608	2650510	 28718e	vmlinux.before
1177974	 254836	1220608	2653418	 287cea	vmlinux.after

Hopefully this is still not a big deal. The per-file trimming with the
static library is not so effective after all.

If fine-grained optimization is desired, some architectures support
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION, which trims dead code per-symbol
basis. When LTO is supported in mainline, even better optimization will
be possible.

Link: https://github.com/ClangBuiltLinux/linux/issues/515Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>
Reported-by: Nkbuild test robot <lkp@intel.com>
Reviewed-by: NNick Desaulniers <ndesaulniers@google.com>

Nobody was opposed to raising minimum GCC version to 4.8 [1]
So, we will drop GCC <= 4.7 support sooner or later.

We always use C++ compiler for building plugins for GCC >= 4.8.

This commit drops the plugin support for GCC <= 4.7 a bit earlier,
which allows us to dump lots of code.

[1] https://lkml.org/lkml/2020/1/23/545Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>
Acked-by: NKees Cook <keescook@chromium.org>

In old days, the "host-progs" syntax was used for specifying host
programs. It was renamed to the current "hostprogs-y" in 2004.

It is typically useful in scripts/Makefile because it allows Kbuild to
selectively compile host programs based on the kernel configuration.

This commit renames like follows:

  always       ->  always-y
  hostprogs-y  ->  hostprogs

So, scripts/Makefile will look like this:

  always-$(CONFIG_BUILD_BIN2C) += ...
  always-$(CONFIG_KALLSYMS)    += ...
      ...
  hostprogs := $(always-y) $(always-m)

I think this makes more sense because a host program is always a host
program, irrespective of the kernel configuration. We want to specify
which ones to compile by CONFIG options, so always-y will be handier.

The "always", "hostprogs-y", "hostprogs-m" will be kept for backward
compatibility for a while.
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

The built-in.a in a sub-directory is created by descending into that
directory. It does not depend on the other sub-directories. Loosen
the dependency.
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

Both 'obj-y += foo/' and 'obj-m += foo/' request Kbuild to visit the
sub-directory foo/, but the difference is that only the former combines
foo/built-in.a into the built-in.a of the current directory because
everything in sub-directories visited by obj-m is supposed to be modular.

So, it makes sense to create built-in.a only if that sub-directory is
reachable by the chain of obj-y. Otherwise, built-in.a will not be
linked into vmlinux anyway. For the same reason, it is pointless to
compile obj-y objects in the directory visited by obj-m.
Signed-off-by: NMasahiro Yamada <masahiroy@kernel.org>

There are both positive and negative options about this feature.
At first, I thought it was a good idea, but actually Linus stated a
negative opinion (https://lkml.org/lkml/2019/9/29/227). I admit it
is ugly and annoying.

The baseline I'd like to keep is the compile-test of uapi headers.
(Otherwise, kernel developers have no way to ensure the correctness
of the exported headers.)

I will maintain a small build rule in usr/include/Makefile.
Remove the other header test functionality.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Since commit 394053f4 ("kbuild: make single targets work more
correctly"), building single targets is really slow.

Speed it up by not descending into unrelated directories.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

KBUILD_SINGLE_TARGETS does not need to contain all the targets.
Change it to keep track the targets only from the current directory
and its subdirectories.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Commit 40df759e ("kbuild: Fix build with binutils <= 2.19")
introduced ar-option and KBUILD_ARFLAGS to deal with old binutils.

According to Documentation/process/changes.rst, the current minimal
supported version of binutils is 2.21 so you can assume the 'D' option
is always supported. Not only GNU ar but also llvm-ar supports it.

With the 'D' option hard-coded, there is no more user of ar-option
or KBUILD_ARFLAGS.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NNick Desaulniers <ndesaulniers@google.com>
Tested-by: NNick Desaulniers <ndesaulniers@google.com>

KBUILD_ENABLE_EXTRA_GCC_CHECKS started as a switch to add extra warning
options for GCC, but now it is a historical misnomer since we use it
also for Clang, DTC, and even kernel-doc.

Rename it to more sensible, shorter KBUILD_EXTRA_WARN.

For the backward compatibility, KBUILD_ENABLE_EXTRA_GCC_CHECKS is still
supported (but not advertised in the documentation).

I also fixed up 'make help', and updated the documentation.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NNathan Chancellor <natechancellor@gmail.com>
Reviewed-by: NNick Desaulniers <ndesaulniers@google.com>
Reviewed-by: NSedat Dilek <sedat.dilek@gmail.com>

Makefile.lib is included by Makefile.modfinal as well as Makefile.build.

Move modkern_cflags to Makefile.lib in order to simplify cmd_cc_o_c
in Makefile.modfinal. Move modkern_cflags as well for consistency.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Add CONFIG_ASM_MODVERSIONS. This allows to remove one if-conditional
nesting in scripts/Makefile.build.

scripts/Makefile.build is run every time Kbuild descends into a
sub-directory. So, I want to avoid $(wildcard ...) evaluation
where possible although computing $(wildcard ...) is so cheap that
it may not make measurable performance difference.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: NGeert Uytterhoeven <geert@linux-m68k.org>

Currently, the single target build directly descends into the directory
of the target. For example,

  $ make foo/bar/baz.o

... directly descends into foo/bar/.

On the other hand, the normal build usually descends one directory at
a time, i.e. descends into foo/, and then foo/bar/.

This difference causes some problems.

[1] miss subdir-asflags-y, subdir-ccflags-y in upper Makefiles

    The options in subdir-{as,cc}flags-y take effect in the current
    and its sub-directories. In other words, they are inherited
    downward. In the example above, the single target will miss
    subdir-{as,cc}flags-y if they are defined in foo/Makefile.

[2] could be built in a different directory

    As Documentation/kbuild/modules.rst section 4.3 says, Kbuild can
    handle files that are spread over several sub-directories.

    The build rule of foo/bar/baz.o may not necessarily be specified in
    foo/bar/Makefile. It might be specifies in foo/Makefile as follows:

    [foo/Makefile]
    obj-y := bar/baz.o

    This often happens when a module is so big that its source files
    are divided into sub-directories.

    In this case, there is no Makefile in the foo/bar/ directory, yet
    the single target descends into foo/bar/, then fails due to the
    missing Makefile. You can still do 'make foo/bar/' for partial
    building, but cannot do 'make foo/bar/baz.s'. I believe the single
    target '%.s' is a useful feature for inspecting the compiler output.

    Some modules work around this issue by putting an empty Makefile
    in every sub-directory.

This commit fixes those problems by making the single target build
descend in the same way as the normal build does.

Another change is the single target build will observe the CONFIG
options. Previously, it allowed users to build the foo.o even when
the corresponding CONFIG_FOO is disabled:

   obj-$(CONFIG_FOO) += foo.o

In the new behavior, the single target build will just fail and show
"No rule to make target ..." (or "Nothing to be done for ..." if the
stale object already exists, but cannot be updated).

The disadvantage of this commit is the build speed. Now that the
single target build visits every directory and parses lots of
Makefiles, it is slower than before. (But, I hope it will not be
too slow.)
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

For the single target building %.symtypes from %.S, $(a_flags) is
expanded into the _KERNEL flags even if the object is a part of a
module.

$(real-obj-m:.o=.symtypes): modkern_aflags := $(KBUILD_AFLAGS_MODULE) $(AFLAGS_MODULE)

... would fix the issue, but it is not nice to duplicate similar code
for every suffix.

Implement modkern_aflags in the same way as modkern_cflags.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Make it even shorter.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>