To simplify the generated lexer, let the hand-made lexer update the
file name and line number for the parser.

I tested this with DEBUG_PARSE, and confirmed the same file names
and line numbers were dumped.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

To simplify the generated lexer, switch to the ASSIGN_VAL state in
the hand-made lexer.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

The lexer has conventionally associated kconf_id data with yylval
to carry additional information to the parser.

No token is relying on this any more.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

T_ENDMENU, T_ENDCHOICE, T_ENDIF are the last users of kconf_id
associated with yylval. Refactor them to not use it.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

In my understanding, special characters such as '.' and '/' are
supported in unquoted words to use bare file paths in the "source"
statement.

With the previous commit surrounding all file paths with double
quotes, we can drop this.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

The Kconfig lexer supports special characters such as '.' and '/' in
the parameter context. In my understanding, the reason is just to
support bare file paths in the source statement.

I do not see a good reason to complicate Kconfig for the room of
ambiguity.

The majority of code already surrounds file paths with double quotes,
and it makes sense since file paths are constant string literals.

Make it treewide consistent now.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: NWolfram Sang <wsa@the-dreams.de>
Acked-by: NGeert Uytterhoeven <geert@linux-m68k.org>
Acked-by: NIngo Molnar <mingo@kernel.org>

I guess this is a constant value instead of a symbol.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: NMichal Simek <michal.simek@xilinx.com>

There is no grammatical ambiguity by using T_WORD for variables.
The parser can distinguish variables from symbols from the context.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Currently, the lexer returns T_ASSIGN for all of =, :=, and +=
associating yylval with the flavor.

I want to make the generated lexer as simple as possible. So, the
lexer should convert keywords to tokens without thinking about the
meaning.

   =  ->  T_EQUAL
  :=  ->  T_COLON_EQUAL
  +=  ->  T_PLUS_EQUAL

Unfortunately, Kconfig uses = instead of == for the equal operator.
So, the same token T_EQUAL is used for assignment and comparison.
The parser can still distinguish them from the context.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

For the keywords "modules", "defconfig_list", and "allnoconfig_y",
the lexer should pass specific tokens instead of generic T_WORD.

This simplifies both the lexer and the parser.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

This commit removes kconf_id::stype to prepare for the entire
removal of kconf_id.c

To simplify the lexer, I want keywords straight-mapped to tokens.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

These are already defined as %left.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Now the comment_stmt is the only user of depends_list. Rename it to
comment_option_list
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Currently, "visible" and "depends on", if defined in a menu entry,
must appear in that order.

The real example is in drivers/media/tuners/Kconfig:

  menu "Customize TV tuners"
          visible if <expr1>
          depends on <expr2>

... is fine, but you cannot change the property order like this:

  menu "Customize TV tuners"
          depends on <expr2>
          visible if <expr1>

Kconfig does not require a specific order of properties. In this case,
menu_add_visibility(() and menu_add_dep() are orthogonal.

Loosen this unreasonable restriction.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

The code block surrounded by "menu" ... "endmenu" is stmt_list.

Remove the redundant menu_block symbol entirely.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

The code block surrounded by "if" ... "endif" is stmt_list.

Remove the redundant if_block symbol entirely.

Remove "stmt_list: stmt_list end" rule as well since it would
obviously cause conflicts.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

This commit decreases 6 shift/reduce conflicts, and finally achieves
conflict-free parser.

Since Kconfig has no terminator for a config block, detecting the end
of config_stmt is not easy.

For example, there are two ways for handling the error in the following
code:

  1 config FOO
  2         =

 [A] Print "unknown option" error, assuming the line 2 is a part of
     config_option_list

 [B] Print "invalid statement", assuming the line 1 is reduced into
     a config_stmt by itself

Bison actually chooses [A] because it performs the shift rather than
the reduction where both are possible.

However, there is no reason to choose one over the other.

Let's remove the option_error, and let it fall back to [B].
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

This commit decreases 15 shift/reduce conflicts.

The location of this error recovery is ambiguous.

For example, there are two ways to interpret the following code:

  1 config FOO
  2         bool "foo"

 [A] Both lines are reduced together into a config_stmt.

 [B] The only line 1 is reduced into a config_stmt, and the line 2
     matches to "option_name error T_EOL"

Of course, we expect [A], but [B] could be grammatically possible.

Kconfig has no terminator for a config block. So, we cannot detect its
end until we see a non-property keyword. People often insert a blank
line between two config blocks, but it is just a coding convention.
Blank lines are actually allowed anywhere in Kconfig files.

The real error is when a property keyword appears right after "endif",
"endchoice", "endmenu",  "source", "comment", or variable assignment.

Instead of fixing the grammatical ambiguity, I chose to simply remove
this error recovery.

The difference is

  unexpected option "bool"

... is turned into a more generic message:

  invalid statement
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

It would be nice to warn if a new line is missing at end of file.

We could do this by checkpatch.pl for arbitrary files, but new line
is rather essential as a statement terminator in Kconfig.

The warning message looks like this:

  kernel/Kconfig.preempt:60:warning: no new line at end of file

Currently, kernel/Kconfig.preempt is the only file with no new line
at end of file. Fix it.

I know there are some false negative cases. For example, no warning
is displayed when the last line contains some whitespaces/comments,
but no new line. Yet, this commit works well for most cases.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

A new file should always start in the INITIAL state.

When the lexer bumps into EOF, the lexer must get back to the INITIAL
state anyway. Remove the redundant <<EOF>> pattern in the PARAM state.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

This commit decreases 8 shift/reduce conflicts.

A certain amount of grammatical ambiguity comes from how to reduce
excessive T_EOL tokens.

Let's take a look at the example code below:

  1  config A
  2          bool "a"
  3
  4          depends on B
  5
  6  config B
  7          def_bool y

The line 3 is melt into "config_option_list", but the line 5 can be
either a part of "config_option_list" or "common_stmt" by itself.

Currently, the lexer converts '\n' to T_EOL verbatim. In Kconfig,
a new line works as a statement terminator, but new lines in empty
lines are not critical since empty lines (or lines that contain only
whitespaces/comments) are just no-op.

If the lexer simply discards no-op lines, the parser will not be
bothered by excessive T_EOL tokens.

Of course, this means we are shifting the complexity from the parser
to the lexer, but it is much easier than tackling on shift/reduce
conflicts.

I introduced the second stage lexer to tweak the behavior.

Discard T_EOL if the previous token is T_EOL or T_HELPTEXT.
Two T_EOL tokens in a row is meaningless. T_HELPTEXT is a special
token that is reduced without T_EOL.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Here, similar matching patters are duplicated in order to look ahead
the '\n' character. If the next character is '\n', the lexer returns
T_WORD_QUOTE because it must be prepared to return T_EOL at the next
match.

Use unput('\n') trick to reduce the code duplication.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Whitespaces are consumed in the COMMAND state anyway.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

All line-oriented statements should be reduced when seeing a T_EOL
token. I guess missing T_EOL for the "visible" statement is just a
mistake. This commit decreases one shift/reduce conflict.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

An unterminated string literal followed by new line is passed to the
parser (with "multi-line strings not supported" warning shown), then
handled properly there.

On the other hand, an unterminated string literal at end of file is
never passed to the parser, then results in memory leak.

[Test Code]

  ----------(Kconfig begin)----------
  source "Kconfig.inc"

  config A
          bool "a"
  -----------(Kconfig end)-----------

  --------(Kconfig.inc begin)--------
  config B
          bool "b\No new line at end of file
  ---------(Kconfig.inc end)---------

[Summary from Valgrind]

  Before the fix:

    LEAK SUMMARY:
       definitely lost: 16 bytes in 1 blocks
       ...

  After the fix:

    LEAK SUMMARY:
       definitely lost: 0 bytes in 0 blocks
       ...

Eliminate the memory leak path by handling this case. Of course, such
a Kconfig file is wrong already, so I will add an error message later.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Currently, warn_ignore_character() displays invalid file name and
line number.

The lexer should use current_file->name and yylineno, while the parser
should use zconf_curname() and zconf_lineno().

This difference comes from that the lexer is always going ahead
of the parser. The parser needs to look ahead one token to make a
shift/reduce decision, so the lexer is requested to scan more text
from the input file.

This commit fixes the warning message from warn_ignored_character().

[Test Code]

  ----(Kconfig begin)----
  /
  -----(Kconfig end)-----

[Output]

  Before the fix:

  <none>:0:warning: ignoring unsupported character '/'

  After the fix:

  Kconfig:1:warning: ignoring unsupported character '/'
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

The only possibility of k_invalid being returned was when
expr_parse_sting() parsed S_OTHER type symbol. This actually never
happened, and this is even clearer since S_OTHER has gone.

Clean up unreachable code.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

The S_OTHER type could be set only when conf_read_simple() is reading
include/config/auto.conf file.

For example, CONFIG_FOO=y exists in include/config/auto.conf but it is
missing from the currently parsed Kconfig files, sym_lookup() allocates
a new symbol, and sets its type to S_OTHER.

Strangely, it will be set to S_STRING by conf_set_sym_val() a few lines
below while it is obviously bool or tristate type. On the other hand,
when CONFIG_BAR="bar" is being dropped from include/config/auto.conf,
its type remains S_OTHER. Because for_all_symbols() omits S_OTHER
symbols, conf_touch_deps() misses to touch include/config/bar.h

This behavior has been a pretty mystery for me, and digging the git
histroy did not help. At least, touching depfiles is broken for string
type symbols.

I removed S_OTHER entirely, and reimplemented it more simply.

If CONFIG_FOO was visible in the previous syncconfig, but is missing
now, what we want to do is quite simple; just call conf_touch_dep()
to touch include/config/foo.h instead of allocating a new symbol data.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

conf_touch_deps() iterates over symbols, touching corresponding
include/config/*.h files as needed.

Split the part that touches a single file into a new helper so it can
be reused.

The new helper, conf_touch_dep(), takes a symbol name as a parameter,
and touches the corresponding include/config/*.h file.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

According to commit 2e3646e5 ("kconfig: integrate split config
into silentoldconfig"), this function was named after split-include
tool, which used to exist in old versions of Linux.

Setting aside the historical reason, rename it into a more intuitive
name. This function touches timestamp files under include/config/
in order to interact with the fixdep tool.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

The two 'goto setsym' statements are reachable only when sym == NULL.

The code below the 'setsym:' label does nothing when sym == NULL
since there is just one if-block guarded by 'if (sym && ...)'.

Hence, 'goto setsym' can be replaced with 'continue'.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

In today's merge_config.sh the order of the config fragment files dictates
the output of a config option. With this approach we will get different
.config files depending on the order of the config fragment files.

So doing something like:
$ ./merge/kconfig/merge_config.sh selftest.config drm.config

Where selftest.config defines DRM=y and drm.config defines DRM=m, the
result will be "DRM=m".

Rework to add a switch to get builtin '=y' precedence over modules '=m',
this will result in "DRM=y". If we do something like this:

$ ./merge/kconfig/merge_config.sh -y selftest.config drm.config
Suggested-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NAnders Roxell <anders.roxell@linaro.org>
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Pull networking fixes from David Miller:
 "One last pull request before heading to Vancouver for LPC, here we have:

   1) Don't forget to free VSI contexts during ice driver unload, from
      Victor Raj.

   2) Don't forget napi delete calls during device remove in ice driver,
      from Dave Ertman.

   3) Don't request VLAN tag insertion of ibmvnic device when SKB
      doesn't have VLAN tags at all.

   4) IPV4 frag handling code has to accomodate the situation where two
      threads try to insert the same fragment into the hash table at the
      same time. From Eric Dumazet.

   5) Relatedly, don't flow separate on protocol ports for fragmented
      frames, also from Eric Dumazet.

   6) Memory leaks in qed driver, from Denis Bolotin.

   7) Correct valid MTU range in smsc95xx driver, from Stefan Wahren.

   8) Validate cls_flower nested policies properly, from Jakub Kicinski.

   9) Clearing of stats counters in mc88e6xxx driver doesn't retain
      important bits in the G1_STATS_OP register causing the chip to
      hang. Fix from Andrew Lunn"

* git://git.kernel.org/pub/scm/linux/kernel/git/davem/net: (41 commits)
  act_mirred: clear skb->tstamp on redirect
  net: dsa: mv88e6xxx: Fix clearing of stats counters
  tipc: fix link re-establish failure
  net: sched: cls_flower: validate nested enc_opts_policy to avoid warning
  net: mvneta: correct typo
  flow_dissector: do not dissect l4 ports for fragments
  net: qualcomm: rmnet: Fix incorrect assignment of real_dev
  net: aquantia: allow rx checksum offload configuration
  net: aquantia: invalid checksumm offload implementation
  net: aquantia: fixed enable unicast on 32 macvlan
  net: aquantia: fix potential IOMMU fault after driver unbind
  net: aquantia: synchronized flow control between mac/phy
  net: smsc95xx: Fix MTU range
  net: stmmac: Fix RX packet size > 8191
  qed: Fix potential memory corruption
  qed: Fix SPQ entries not returned to pool in error flows
  qed: Fix blocking/unlimited SPQ entries leak
  qed: Fix memory/entry leak in qed_init_sp_request()
  inet: frags: better deal with smp races
  net: hns3: bugfix for not checking return value
  ...

Merge tag 'kbuild-fixes-v4.20' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild

Pull Kbuild fixes from Masahiro Yamada:

 - fix build errors in binrpm-pkg and bindeb-pkg targets

 - fix false positive matches in merge_config.sh

 - fix build version mismatch in deb-pkg target

 - fix dtbs_install handling in (bin)deb-pkg target

 - revert a commit that allows setlocalversion to write to source tree

* tag 'kbuild-fixes-v4.20' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild:
  builddeb: Fix inclusion of dtbs in debian package
  Revert "scripts/setlocalversion: git: Make -dirty check more robust"
  kbuild: deb-pkg: fix too low build version number
  kconfig: merge_config: avoid false positive matches from comment lines
  kbuild: deb-pkg: fix bindeb-pkg breakage when O= is used
  kbuild: rpm-pkg: fix binrpm-pkg breakage when O= is used

Pull btrfs fixes from David Sterba:
 "Several fixes to recent release (4.19, fixes tagged for stable) and
  other fixes"

* tag 'for-4.20-rc1-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux:
  Btrfs: fix missing delayed iputs on unmount
  Btrfs: fix data corruption due to cloning of eof block
  Btrfs: fix infinite loop on inode eviction after deduplication of eof block
  Btrfs: fix deadlock on tree root leaf when finding free extent
  btrfs: avoid link error with CONFIG_NO_AUTO_INLINE
  btrfs: tree-checker: Fix misleading group system information
  Btrfs: fix missing data checksums after a ranged fsync (msync)
  btrfs: fix pinned underflow after transaction aborted
  Btrfs: fix cur_offset in the error case for nocow

Pull ext4 fixes from Ted Ts'o:
 "A large number of ext4 bug fixes, mostly buffer and memory leaks on
  error return cleanup paths"

* tag 'ext4_for_linus_stable' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4:
  ext4: missing !bh check in ext4_xattr_inode_write()
  ext4: fix buffer leak in __ext4_read_dirblock() on error path
  ext4: fix buffer leak in ext4_expand_extra_isize_ea() on error path
  ext4: fix buffer leak in ext4_xattr_move_to_block() on error path
  ext4: release bs.bh before re-using in ext4_xattr_block_find()
  ext4: fix buffer leak in ext4_xattr_get_block() on error path
  ext4: fix possible leak of s_journal_flag_rwsem in error path
  ext4: fix possible leak of sbi->s_group_desc_leak in error path
  ext4: remove unneeded brelse call in ext4_xattr_inode_update_ref()
  ext4: avoid possible double brelse() in add_new_gdb() on error path
  ext4: avoid buffer leak in ext4_orphan_add() after prior errors
  ext4: avoid buffer leak on shutdown in ext4_mark_iloc_dirty()
  ext4: fix possible inode leak in the retry loop of ext4_resize_fs()
  ext4: fix missing cleanup if ext4_alloc_flex_bg_array() fails while resizing
  ext4: add missing brelse() update_backups()'s error path
  ext4: add missing brelse() add_new_gdb_meta_bg()'s error path
  ext4: add missing brelse() in set_flexbg_block_bitmap()'s error path
  ext4: avoid potential extra brelse in setup_new_flex_group_blocks()

Pull x86 fixes from Thomas Gleixner:
 "A set of x86 fixes:

   - Cure the LDT remapping to user space on 5 level paging which ended
     up in the KASLR space

   - Remove LDT mapping before freeing the LDT pages

   - Make NFIT MCE handling more robust

   - Unbreak the VSMP build by removing the dependency on paravirt ops

   - Support broken PIT emulation on Microsoft hyperV

   - Don't trace vmware_sched_clock() to avoid tracer recursion

   - Remove -pipe from KBUILD CFLAGS which breaks clang and is also
     slower on GCC

   - Trivial coding style and typo fixes"

* 'x86-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  x86/cpu/vmware: Do not trace vmware_sched_clock()
  x86/vsmp: Remove dependency on pv_irq_ops
  x86/ldt: Remove unused variable in map_ldt_struct()
  x86/ldt: Unmap PTEs for the slot before freeing LDT pages
  x86/mm: Move LDT remap out of KASLR region on 5-level paging
  acpi/nfit, x86/mce: Validate a MCE's address before using it
  acpi/nfit, x86/mce: Handle only uncorrectable machine checks
  x86/build: Remove -pipe from KBUILD_CFLAGS
  x86/hyper-v: Fix indentation in hv_do_fast_hypercall16()
  Documentation/x86: Fix typo in zero-page.txt
  x86/hyper-v: Enable PIT shutdown quirk
  clockevents/drivers/i8253: Add support for PIT shutdown quirk

Pull perf fixes from Thomas Gleixner:
 "A bunch of perf tooling fixes:

   - Make the Intel PT SQL viewer more robust

   - Make the Intel PT debug log more useful

   - Support weak groups in perf record so it's behaving the same way as
     perf stat

   - Display the LBR stats in callchain entries properly in perf top

   - Handle different PMu names with common prefix properlin in pert
     stat

   - Start syscall augmenting in perf trace. Preparation for
     architecture independent eBPF instrumentation of syscalls.

   - Fix build breakage in JVMTI perf lib

   - Fix arm64 tools build failure wrt smp_load_{acquire,release}"

* 'perf-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  perf tools: Do not zero sample_id_all for group members
  perf tools: Fix undefined symbol scnprintf in libperf-jvmti.so
  perf beauty: Use SRCARCH, ARCH=x86_64 must map to "x86" to find the headers
  perf intel-pt: Add MTC and CYC timestamps to debug log
  perf intel-pt: Add more event information to debug log
  perf scripts python: exported-sql-viewer.py: Fix table find when table re-ordered
  perf scripts python: exported-sql-viewer.py: Add help window
  perf scripts python: exported-sql-viewer.py: Add Selected branches report
  perf scripts python: exported-sql-viewer.py: Fall back to /usr/local/lib/libxed.so
  perf top: Display the LBR stats in callchain entry
  perf stat: Handle different PMU names with common prefix
  perf record: Support weak groups
  perf evlist: Move perf_evsel__reset_weak_group into evlist
  perf augmented_syscalls: Start collecting pathnames in the BPF program
  perf trace: Fix setting of augmented payload when using eBPF + raw_syscalls
  perf trace: When augmenting raw_syscalls plug raw_syscalls:sys_exit too
  perf examples bpf: Start augmenting raw_syscalls:sys_{start,exit}
  tools headers barrier: Fix arm64 tools build failure wrt smp_load_{acquire,release}

Pull timer fix from Thomas Gleixner:
 "Just the removal of a redundant call into the sched deadline overrun
  check"

* 'timers-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  posix-cpu-timers: Remove useless call to check_dl_overrun()