Add 'const' where a function does not write to the pointer dereferenes.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

The callers of this function expect (unsigned char *). I do not see
a good reason to make this function return (void *).
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

You can do equivalent things with strspn(). I do not see noticeable
performance difference.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

sym_entry::sym is (unsigned char *) instead of (char *) because
kallsyms exploits the MSB for compression, and the characters are
used as the index of token_profit array.

However, it requires casting (unsigned char *) to (char *) in some
places since standard library functions such as strcmp(), strlen()
expect (char *).

Introduce a new helper, sym_name(), which advances the given pointer
by 1 and casts it to (char *).
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

l <= strlen(sym_name) is unnecessary for prefix matching.
strncmp() will do.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Since commit 6f00df24 ("[PATCH] Strip local symbols from kallsyms"),
all symbols starting '$' are ignored.

is_arm_mapping_symbol() particularly ignores $a, $t, etc. but it is
redundant.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Currently, record_relative_base() iterates over the entire table to
find the minimum address, but it is not efficient because we sort
the table anyway.

After sort_symbol(), the table is sorted by address. (kallsyms parses
the 'nm -n' output, so the data is already sorted by address, but this
commit does not rely on it.)

Move record_relative_base() after sort_symbols(), and take the first
non-absolute symbol value.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Currently, build_initial_tok_table() trims unused symbols, but it is
called after sort_symbols().

It is not efficient to sort the huge table that contains unused entries.
Shrink the table before sorting it.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

build_initial_tok_table() overwrites unused sym_entry to shrink the
table size. Before the entry is overwritten, table[i].sym must be freed
since it is malloc'ed data.

This fixes the 'definitely lost' report from valgrind. I ran valgrind
against x86_64_defconfig of v5.4-rc8 kernel, and here is the summary:

[Before the fix]

  LEAK SUMMARY:
     definitely lost: 53,184 bytes in 2,874 blocks

[After the fix]

  LEAK SUMMARY:
     definitely lost: 0 bytes in 0 blocks
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

This is not defined in the standard headers. #ifndef is unneeded.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

When 'exported twice' is warned, let sym_add_exported() return without
updating the symbol info. This respects the previous export, which is
ordered first in modules.order

This simplifies the code too.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Now that there is no overwrap between symbols from ELF files and
ones from Module.symvers.

So, the 'exported twice' warning should be reported irrespective
of where the symbol in question came from.

The exceptional case is external module; in some cases, we build
an external module to provide a different version/variant of the
corresponding in-kernel module, overriding the same set of exported
symbols.

You can see this use-case in upstream; tools/testing/nvdimm/libnvdimm.ko
replaces drivers/nvdimm/libnvdimm.ko in order to link it against mocked
version of core kernel symbols.

So, let's relax the 'exported twice' warning when building external
modules. The multiple export from external modules is warned only
when the previous one is from vmlinux or itself.

With this refactoring, the ugly preloading goes away.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

It is complicated to add mocked-up symbols for pre-handling CRC.
Handle CRC after all the export symbols in the relevant module
are registered.

Call handle_modversion() after the handle_symbol() iteration.

In some cases, I see atand-alone __crc_* without __ksymtab_*.
For example, ARCH=arm allyesconfig produces __crc_ccitt_veneer and
__crc_itu_t_veneer. I guess they come from crc_ccitt, crc_itu_t,
respectively. Since __*_veneer are auto-generated symbols, just
ignore them.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

This function handles not only modversions, but also unresolved
symbols, export symbols, etc.

Rename it to a more proper function name.

While I was here, I also added the 'const' qualifier to *sym.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Currently, namespace_from_kstrtabns() relies on the fact that
namespace strings are recorded in the __ksymtab_strings section.
Actually, it is coded in include/linux/export.h, but modpost does
not need to hard-code the section name.

Elf_Sym::st_shndx holds the index of the relevant section. Using it is
a more portable way to get the namespace string.

Make namespace_from_kstrtabns() simply call sym_get_data(), and delete
the info->ksymtab_strings .

While I was here, I added more 'const' qualifiers to pointers.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

When CONFIG_MODULE_REL_CRCS is enabled, the value of __crc_* is not
an absolute value, but the address to the CRC data embedded in the
.rodata section.

Getting the data pointed by the symbol value is somewhat complex.
Split it out into a new helper, sym_get_data().

I will reuse it to refactor namespace_from_kstrtabns() in the next
commit.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Currently, some sanity checks for uapi headers are done by
scripts/headers_check.pl, which is wired up to the 'headers_check'
target in the top Makefile.

It is true compiling headers has better test coverage, but there
are still several headers excluded from the compile test. I like
to keep headers_check.pl for a while, but we can delete a lot of
code by moving the build rule to usr/include/Makefile.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

There are both positive and negative options about this feature.
At first, I thought it was a good idea, but actually Linus stated a
negative opinion (https://lkml.org/lkml/2019/9/29/227). I admit it
is ugly and annoying.

The baseline I'd like to keep is the compile-test of uapi headers.
(Otherwise, kernel developers have no way to ensure the correctness
of the exported headers.)

I will maintain a small build rule in usr/include/Makefile.
Remove the other header test functionality.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

GNU Make manual says:

  $?
      The names of all the prerequisites that are newer than the target,
      with spaces between them.

To reflect this, rename any-prereq to newer-prereqs, which is clearer
and more intuitive.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

The incremental build of Linux kernel is pretty slow when lots of
objects are compiled. The rebuild of allmodconfig may take a few
minutes even when none of the objects needs to be rebuilt.

The time-consuming part in the incremental build is the evaluation of
if_changed* macros since they are used in the recipes to compile C and
assembly source files into objects.

I notice the following code in if_changed* is expensive:

  $(filter-out $(PHONY) $(wildcard $^),$^)

In the incremental build, every object has its .*.cmd file, which
contains the auto-generated list of included headers. So, $^ are
expanded into the long list of the source file + included headers,
and $(wildcard $^) checks whether they exist.

It may not be clear why this check exists there.

Here is the record of my research.

[1] The first code addition into Kbuild

This code dates back to 2002. It is the pre-git era. So, I copy-pasted
it from the historical git tree.

| commit 4a6db0791528c220655b063cf13fefc8470dbfee (HEAD)
| Author: Kai Germaschewski <kai@tp1.ruhr-uni-bochum.de>
| Date:   Mon Jun 17 00:22:37 2002 -0500
|
|     kbuild: Handle removed headers
|
|     New and old way to handle dependencies would choke when a file
|     #include'd by other files was removed, since the dependency on it was
|     still recorded, but since it was gone, make has no idea what to do about
|     it (and would complain with "No rule to make <file> ...")
|
|     We now add targets for all the previously included files, so make will
|     just ignore them if they disappear.
|
| diff --git a/Rules.make b/Rules.make
| index 6ef827d3df39..7db5301ea7db 100644
| --- a/Rules.make
| +++ b/Rules.make
| @@ -446,7 +446,7 @@ if_changed = $(if $(strip $? \
|  # execute the command and also postprocess generated .d dependencies
|  # file
|
| -if_changed_dep = $(if $(strip $? \
| +if_changed_dep = $(if $(strip $? $(filter-out FORCE $(wildcard $^),$^)\
|                           $(filter-out $(cmd_$(1)),$(cmd_$@))\
|                           $(filter-out $(cmd_$@),$(cmd_$(1)))),\
|         @set -e; \
| diff --git a/scripts/fixdep.c b/scripts/fixdep.c
| index b5d7bee8efc7..db45bd1888c0 100644
| --- a/scripts/fixdep.c
| +++ b/scripts/fixdep.c
| @@ -292,7 +292,7 @@ void parse_dep_file(void *map, size_t len)
|                 exit(1);
|         }
|         memcpy(s, m, p-m); s[p-m] = 0;
| -       printf("%s: \\\n", target);
| +       printf("deps_%s := \\\n", target);
|         m = p+1;
|
|         clear_config();
| @@ -314,7 +314,8 @@ void parse_dep_file(void *map, size_t len)
|                 }
|                 m = p + 1;
|         }
| -       printf("\n");
| +       printf("\n%s: $(deps_%s)\n\n", target, target);
| +       printf("$(deps_%s):\n", target);
|  }
|
|  void print_deps(void)

The "No rule to make <file> ..." error can be solved by passing -MP to
the compiler, but I think the detection of header removal is a good
feature. When a header is removed, all source files that previously
included it should be re-compiled. This makes sure we has correctly
got rid of #include directives of it.

This is also related with the behavior of $?. The GNU Make manual says:

  $?
      The names of all the prerequisites that are newer than the target,
      with spaces between them.

This does not explain whether a non-existent prerequisite is considered
to be newer than the target.

At this point of time, GNU Make 3.7x was used, where the $? did not
include non-existent prerequisites. Therefore,

  $(filter-out FORCE $(wildcard $^),$^)

was useful to detect the header removal, and to rebuild the related
objects if it is the case.

[2] Change of $? behavior

Later, the behavior of $? was changed (fixed) to include prerequisites
that did not exist.

First, GNU Make commit 64e16d6c00a5 ("Various changes getting ready for
the release of 3.81.") changed it, but in the release test of 3.81, it
turned out to break the kernel build.

See these:

 - http://lists.gnu.org/archive/html/bug-make/2006-03/msg00003.html
 - https://savannah.gnu.org/bugs/?16002
 - https://savannah.gnu.org/bugs/?16051

Then, GNU Make commit 6d8d9b74d9c5 ("Numerous updates to tests for
issues found on Cygwin and Windows.") reverted it for the 3.81 release
to give Linux kernel time to adjust to the new behavior.

After the 3.81 release, GNU Make commit 7595f38f62af ("Fixed a number
of documentation bugs, plus some build/install issues:") re-added it.

[3] Adjustment to the new $? behavior on Kbuild side

Meanwhile, the kernel build was changed by commit 4f193362 ("kbuild:
change kbuild to not rely on incorrect GNU make behavior") to adjust to
the new $? behavior.

[4] GNU Make 3.82 released in 2010

GNU Make 3.82 was the first release that integrated the correct $?
behavior. At this point, Kbuild dealt with GNU Make versions with
different $? behaviors.

 3.81 or older:
    $? does not contain any non-existent prerequisite.
    $(filter-out $(PHONY) $(wildcard $^),$^) was useful to detect
    removed include headers.

 3.82 or newer:
    $? contains non-existent prerequisites. When a header is removed,
    it appears in $?. $(filter-out $(PHONY) $(wildcard $^),$^) became
    a redundant check.

With the correct $? behavior, we could have dropped the expensive
check for 3.82 or later, but we did not. (Maybe nobody noticed this
optimization.)

[5] The .SECONDARY special target trips up $?

Some time later, I noticed $? did not work as expected under some
circumstances. As above, $? should contain non-existent prerequisites,
but the ones specified as SECONDARY do not appear in $?.

I asked this in GNU Make ML, and it seems a bug:

  https://lists.gnu.org/archive/html/bug-make/2019-01/msg00001.html

Since commit 8e9b61b2 ("kbuild: move .SECONDARY special target to
Kbuild.include"), all files, including headers listed in .*.cmd files,
are treated as secondary.

So, we are back into the incorrect $? behavior.

If we Kbuild want to react to the header removal, we need to keep
$(filter-out $(PHONY) $(wildcard $^),$^) but this makes the rebuild
so slow.

[Summary]

 - I believe noticing the header removal and recompiling related objects
   is a nice feature for the build system.

 - If $? worked correctly, $(filter-out $(PHONY),$?) would be enough
   to detect the header removal.

 - Currently, $? does not work correctly when used with .SECONDARY,
   and Kbuild is hit by this bug.

 - I filed a bug report for this, but not fixed yet as of writing.

 - Currently, the header removal is detected by the following expensive
   code:

    $(filter-out $(PHONY) $(wildcard $^),$^)

 - I do not want to revert commit 8e9b61b2 ("kbuild: move
   .SECONDARY special target to Kbuild.include"). Specifying
   .SECONDARY globally is clean, and it matches to the Kbuild policy.

This commit proactively removes the expensive check since it makes the
incremental build faster. A downside is Kbuild will no longer be able
to notice the header removal.

You can confirm it by the full-build followed by a header removal, and
then re-build.

  $ make defconfig all
    [ full build ]
  $ rm include/linux/device.h
  $ make
    CALL    scripts/checksyscalls.sh
    CALL    scripts/atomic/check-atomics.sh
    DESCEND  objtool
    CHK     include/generated/compile.h
  Kernel: arch/x86/boot/bzImage is ready  (#11)
    Building modules, stage 2.
    MODPOST 12 modules

Previously, Kbuild noticed a missing header and emits a build error.
Now, Kbuild is fine with it. This is an unusual corner-case, not a big
deal. Once the $? bug is fixed in GNU Make, everything will work fine.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

The local variable, ns_entry, is unneeded.

While I was here, I also cleaned up the comparison with NULL or 0.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NMatthias Maennich <maennich@google.com>

scripts/nsdeps is written to take care of only in-tree modules.
Perhaps, this is not a bug, but just a design. At least,
Documentation/core-api/symbol-namespaces.rst focuses on in-tree modules.

Having said that, some people already tried nsdeps for external modules.
So, it would be nice to support it.
Reported-by: NSteve French <smfrench@gmail.com>
Reported-by: NJessica Yu <jeyu@kernel.org>
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Tested-by: NJessica Yu <jeyu@kernel.org>
Acked-by: NJessica Yu <jeyu@kernel.org>
Reviewed-by: NMatthias Maennich <maennich@google.com>
Tested-by: NMatthias Maennich <maennich@google.com>

The modpost, with the -d option given, generates per-module .ns_deps
files.

Kbuild generates per-module .mod files to carry module information.
This is convenient because Make handles multiple jobs in parallel
when the -j option is given.

On the other hand, the modpost always runs as a single thread.
I do not see a strong reason to produce separate .ns_deps files.

This commit changes the modpost to generate just one file,
modules.nsdeps, each line of which has the following format:

  <module_name>: <list of missing namespaces>

Please note it contains *missing* namespaces instead of required ones.
So, modules.nsdeps is empty if the namespace dependency is all good.

This will work more efficiently because spatch will no longer process
already imported namespaces. I removed the '(if needed)' from the
nsdeps log since spatch is invoked only when needed.

This also solves the stale .ns_deps problem reported by Jessica Yu:

  https://lkml.org/lkml/2019/10/28/467Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Tested-by: NJessica Yu <jeyu@kernel.org>
Acked-by: NJessica Yu <jeyu@kernel.org>
Reviewed-by: NMatthias Maennich <maennich@google.com>
Tested-by: NMatthias Maennich <maennich@google.com>

buf_write() allocates memory. Free it.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

'make nsdeps' invokes the modpost three times at most; before linking
vmlinux, before building modules, and finally for generating .ns_deps
files. Running the modpost again and again is not efficient.

The last two can be unified. When the -d option is given, the modpost
still does the usual job, and in addition, generates .ns_deps files.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Tested-by: NMatthias Maennich <maennich@google.com>
Reviewed-by: NMatthias Maennich <maennich@google.com>

Signed-off-by: NBhaskar Chowdhury <unixbhaskar@gmail.com>
Acked-by: NAlexander Kapshuk <alexander.kapshuk@gmail.com>
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

If ncurses is installed, but at a non-default location, the previous
error message was not helpful in resolving the situation.  Now it will
suggest that pkg-config might need to be installed in addition to
ncurses.
Signed-off-by: NAlyssa Ross <hi@alyssa.is>
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

make listnewconfig will list the individual options that need to be set.
This is useful but there's no easy way to get the help text associated
with the options at the same time. Introduce a new targe
'make helpnewconfig' which lists the full help text of all the
new options as well. This makes it easier to automatically generate
changes that are easy for humans to review. This command also adds
markers between each option for easier parsing.
Signed-off-by: NLaura Abbott <labbott@redhat.com>
Acked-by: NRandy Dunlap <rdunlap@infradead.org>
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Add a 'dir-pkg' target which just creates the same directory structures
as in tar-pkg, but doesn't package anything.
Useful when the user wants to copy the kernel tree on a machine using
ssh, rsync or whatever.
Signed-off-by: NMatteo Croce <mcroce@redhat.com>
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Some "make help" text lines extend beyond 80 characters.
Wrap them before an opening parenthesis, or before 80 characters.
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

This patch replaces backquote to dollar parenthesis syntax for better
readability.
Signed-off-by: NBhaskar Chowdhury <unixbhaskar@gmail.com>
Acked-by: NRandy Dunlap <rdunlap@infradead.org>
Acked-by: NNico Schottelius <nico-linuxsetlocalversion@schottelius.org>
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Since commit 394053f4 ("kbuild: make single targets work more
correctly"), building single targets is really slow.

Speed it up by not descending into unrelated directories.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

KBUILD_SINGLE_TARGETS does not need to contain all the targets.
Change it to keep track the targets only from the current directory
and its subdirectories.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Since commit 040fcc81 ("kbuild: improved modversioning support for
external modules"), the external module build reads Module.symvers in
the directory of the module itself, then dumps symbols back into it.
It accumulates stale symbols in the file when you build an external
module incrementally.

The idea behind it was, as the commit log explained, you can copy
Modules.symvers from one module to another when you need to pass symbol
information between two modules. However, the manual copy of the file
sounds questionable to me, and containing stale symbols is a downside.

Some time later, commit 0d96fb20 ("kbuild: Add new Kbuild variable
KBUILD_EXTRA_SYMBOLS") introduced a saner approach.

So, this commit removes the former one. Going forward, the external
module build dumps symbols into Module.symvers to be carried via
KBUILD_EXTRA_SYMBOLS, but never reads it automatically.

With the -I option removed, there is no one to set the external_module
flag unless KBUILD_EXTRA_SYMBOLS is passed. Now the -i option does it
instead.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

When building external modules, $(objtree)/Module.symvers is scanned
for symbol information of vmlinux and in-tree modules.

Additionally, vmlinux is parsed if it exists in $(objtree)/.
This is totally redundant since all the necessary information is
contained in $(objtree)/Module.symvers.

Do not parse vmlinux at all for external module builds. This makes
sense because vmlinux is deleted by 'make clean'.

'make clean' leaves all the build artifacts for building external
modules. vmlinux is unneeded for that.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

The comment line "When building external modules ..." explains
the same thing as "Include the module's Makefile ..." a few lines
below.

The comment "they may be used when building the .mod.c file" is no
longer true; .mod.c file is compiled in scripts/Makefile.modfinal
since commit 9b9a3f20 ("kbuild: split final module linking out
into Makefile.modfinal"). I still keep the code in case $(obj) or
$(src) is used in the external module Makefile.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

util.c exists both in scripts/kconfig/ and scripts/kconfig/lxdialog.

Prior to commit 54b8ae66 ("kbuild: change *FLAGS_<basetarget>.o
to take the path relative to $(obj)"), Kbuild could not pass different
flags to source files with the same basename. Now that this issue
was solved, you can split util.c out of parser.y and compile them
independently of each other.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

This tool is only used by drivers/video/logo/Makefile. No reason to
keep it in scripts/.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

gcc's -freorder-blocks-and-partition option makes it group frequently
and infrequently used code in .text.hot and .text.unlikely sections
respectively.  At least when building modules on s390, this option is
used by default.

gdb assumes that all code is located in .text section, and that .text
section is located at module load address.  With such modules this is no
longer the case: there is code in .text.hot and .text.unlikely, and
either of them might precede .text.

Fix by explicitly telling gdb the addresses of code sections.

It might be tempting to do this for all sections, not only the ones in
the white list.  Unfortunately, gdb appears to have an issue, when
telling it about e.g. loadable .note.gnu.build-id section causes it to
think that non-loadable .note.Linux section is loaded at address 0,
which in turn causes NULL pointers to be resolved to bogus symbols.  So
keep using the white list approach for the time being.

Link: http://lkml.kernel.org/r/20191028152734.13065-1-iii@linux.ibm.comSigned-off-by: NIlya Leoshkevich <iii@linux.ibm.com>
Reviewed-by: NJan Kiszka <jan.kiszka@siemens.com>
Cc: Kieran Bingham <kbingham@kernel.org>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

The nsdeps script passes a list of the module source files to
generate_deps_for_ns() as a space delimited string named $mod_source_files,
which then passes it to spatch. But since $mod_source_files is not encased
in quotes, each source file in that string is treated as a separate shell
function argument (as $2, $3, $4, etc.).  However, the spatch invocation
only refers to $2, so only the first file out of $mod_source_files is
processed by spatch.

This causes problems (namely, the MODULE_IMPORT_NS() statement doesn't
get inserted) when a module is composed of many source files and the
"main" module file containing the MODULE_LICENSE() statement is not the
first file listed in $mod_source_files. Fix this by encasing
$mod_source_files in quotes so that the entirety of the string is
treated as a single argument and can be referred to as $2.

In addition, put quotes in the variable assignment of mod_source_files
to prevent any shell interpretation and field splitting.
Reviewed-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: NMatthias Maennich <maennich@google.com>
Signed-off-by: NJessica Yu <jeyu@kernel.org>