We mostly move config options from board header files to Kconfig,
but sometimes config defines come from CONFIG_SYS_EXTRA_OPTIONS.

Historically, CONFIG_SYS_EXTRA_OPTIONS originates in boards.cfg,
which was used as a central database of configuration prior to the
Kconfig conversion.

Now, we want to migrate to primary entries in Kconfig rather than
option list in CONFIG_SYS_EXTRA_OPTIONS, so it should be helpful to
have the tool to cleanup CONFIG_SYS_EXTRA_OPTIONS automatically.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NTom Rini <trini@konsulko.com>

I want to reuse this routine in the next commit.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NTom Rini <trini@konsulko.com>

Correct the clean-up of such defines that continue across multiple
lines, like follows:

  #define CONFIG_FOO "this continues to the next line " \
          "this line should be removed too" \
          "this line should be removed as well"
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NTom Rini <trini@konsulko.com>

Show code diff in color if --color option is given.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NTom Rini <trini@konsulko.com>

The header cleanup feature of this tool now removes empty ifdef's,
successive blank lines as well as moved option defines.  So, we
want to see a little more context to check which lines were deleted.

It is true that we can see it by "git diff", but it would not work
in the --dry-run mode.  So, here, this commit.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

The tools/moveconfig.py has a feature to cleanup #define/#undef's
of moved config options, but I want this tool to do a better job.

For example, when we are moving CONFIG_FOO and its define is
surrounded by #ifdef ... #endif, like follows:

  #ifdef CONFIG_BAR
  #  define CONFIG_FOO
  #endif

The header cleanup will leave empty #ifdef ... #endif:

  #ifdef CONFIG_BAR
  #endif

Likewise, if a define line between two blank lines

  <blank line>
  #define CONFIG_FOO
  <blank lines.

... is deleted, the result of the clean-up will be successive empty
lines, which is a coding-style violation.

It is tedious to remove left-over garbage lines manually, so I want
the tool to take care of this.  The tool's job is still not perfect,
so we should check the output of the tool, but I hope our life will
be much easier with this patch.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

The clean tree (make mrproper) and compilers are required when moving
config options, but not needed when we only cleanup headers.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NTom Rini <trini@konsulko.com>

The files in include/generated are generated during build and removed
by "make mrproper", so it has no point to touch them by this tool.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NTom Rini <trini@konsulko.com>

There are some cases where config options are moved, but they are
ripped off at the final savedefconfig stage:

  - The moved option is not user-configurable, for example, due to
    a missing prompt in the Kconfig entry

  - The config was not defined in the original config header despite
    the Kconfig specifies it as non-bool type

  - The config define in the header contains reference to another
    macro, for example:
        #define CONFIG_CONS_INDEX     (CONFIG_SYS_LPC32XX_UART - 2)
    The current moveconfig does not support recursive macro expansion.

In these cases, the conversion is very likely to be an unexpected
result.  That is why I decided to display the log in yellow color
in commit 5da4f857 ("tools: moveconfig: report when CONFIGs are
removed by savedefconfig").

It would be nice to display the list of suspicious boards when the
tool finishes processing.  It is highly recommended to check the
defconfigs once again when this message is displayed.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

Since commit 1d085568 ("tools: moveconfig: display log atomically
in more readable format"), the function color_text() is clever enough
to exclude LF from escape sequences.  Exploit it for removing the
"for" loops from Slots.show_failed_boards().

Also, display "(the list has been saved in moveconfig.failed)" if
there are failed boards.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

The subprocess.Popen() does not change the child process's working
directory if cwd=None is given.  Let's exploit this fact to refactor
the source directory handling.

We no longer have to pass "-C <reference_src_dir>" to the sub-process
because self.current_src_dir tracks the source tree against which we
want to run defconfig/autoconf.

The flag self.use_git_ref is not necessary either because we can know
the current state by checking whether the self.current_src_dir is a
valid string or None.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

The class WorkDir can be used in a very generic way, but currently
it is only used for containing a reference source directory.

This commit changes it for a more dedicated use.  The move_config
function can be more readable by enclosing the git-clone and git-
checkout in the class constructor.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

When moving an integer type option with default value 1, the tool
moves configs with the same value as the default (, and then removed
by the later savedefconfig).  This is a needless operation.

The KconfigParser.parse_one_config() should compare the config after
the "=y -> =1" fixup.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

This option allows the 'make autoconf.mk' step to run against a former
repo state, while the savedefconfig step runs against the current repo
state. This is convenient for the case where something in the Kconfig
has changed such that the defconfig is no longer complete with the new
Kconfigs. This feature allows the .config to be built assuming those old
Kconfigs, but then savedefconfig based on the new state of the Kconfigs.

If in doubt, always specify this switch. It will always do the right
thing even if not required, but if it was required and you don't use it,
the moved configs will be incorrect. When not using this switch, you
must very carefully evaluate that all moved configs are correct.
Signed-off-by: NJoe Hershberger <joe.hershberger@ni.com>
Reviewed-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

The old color blends in with similar messages and makes them not stand
out.
Signed-off-by: NJoe Hershberger <joe.hershberger@ni.com>
Reviewed-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

Signed-off-by: NJoe Hershberger <joe.hershberger@ni.com>
Reviewed-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

The Slot.poll() method is already complicated and a new feature
we are going to add will make it more difficult to understand
the execution flow.

Refactor it with helper methods, .handle_error(), .do_defconfig(),
.do_autoconf(), .do_savedefconfig, and .update_defconfig().
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>

I found "tools/moveconfig -s" might be useful for defconfig re-sync.
I could optimize it for re-sync if I wanted, but I do not want to
make the code complex for this feature.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

Now, this tools invokes "make savedefconfig" only when it needs to
do so, but there might be cases where a user wants the tool to do
savedefconfig forcibly, for example, some defconfigs were already
out of sync and the user wants to fix it as well.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

There are various factors that determine if the given defconfig is
updated, and it is probably what users are more interested in.

Show the log when the defconfig is updated.  Also, copy the file
only when the file content was really updated to avoid changing
the time stamp needlessly.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

This is a rare case, but there is still possibility that some CONFIG
is moved to the .config, but it is removed by "make savedefconfig".
(For example, it happens when the specified CONFIG has no prompt in
the Kconfig entry, i.e. it is not user-configurable.)

It might be an unexpected case.  So, display the log in this case
(in yellow color to gain user's attention if --color option is given).
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

Now, "make savedefconfig" does not always happen.  Display the log
when it happens.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

If no CONFIG option is moved to the .config, no need to sync the
defconfig file.  This accelerates the processing by skipping
needless "make savedefconfig".
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

Move similar code to finish() function.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

Before this commit, the log was displayed in the format:

<defconfig_name>   : <action1>
<defconfig_name>   : <action2>
<defconfig_name>   : <action3>

When we move multiple CONFIGs at the same time, we see as many
<defconfig_name> strings as actions for every defconfig, which is
redundant information.

Moreover, since normal log and error log are displayed separately,
Messages from different threads could be mixed, like this:

<foo>              : <action1>
<foo>              : <action2>
<bar>              : <action1>
<bar>              : <action2>
<foo>              : <error_log>

This commit makes sure to call "print" once a defconfig, which
enables atomic logging for each defconfig.  It also makes it
possible to refactor the log format as follows:

<foo_defconfig>
    <action1>
    <action2>
    <error_log>

<bar_defconfig>
    <action1>
    <action2>
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

This will help further improvement/clean-up.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

The paths to .config, include/autoconf.mk, include/config/auto.conf
are not changed during the defconfig walk.  Compute them only once
when a new class instance is created.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

We still pass the input file with CONFIG name, type, default value
in each line, but the last two fields are just ignored by the tool.
So, let's deprecate the input file and allow users to give CONFIG
names directly from the command line.  The types and default values
are automatically detected and handled nicely by the tool.

Going forward, we can use this tool more easily like:

  tools/moveconfig.py CONFIG_FOO CONFIG_BAR

Update the documentation and fix some typos I noticed while I was
working on.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

Now types and defalut values given by the input file are just
ignored.  Delete unnecessary code.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

Commit 96464bad ("moveconfig: Always run savedefconfig on the
moved config") changed the work flow of this tool a lot from the
original intention when this tool was designed first.

Since then, before running this tool, users must edit the Kconfig to
add the menu entries for the configs they are moving.  It means users
had already specified the type and the default value for each CONFIG
via its Kconfig entry.  Nevertheless, users are still required to
dictate the same type and the default value in the input file.  This
is tedious to use.  So, my idea here is to deprecate the latter.

Before moving forward with it, there is one issue worth mentioning;
since the savedefconfig re-sync was introduced, this tool has not
been able to move bool options with "default y".  Joe sent a patch
to solve this problem about a year ago, but it was not applied for
some reasons.  Now, he came back with an updated patch, so this
problem will be fixed soon.

For other use cases, I see no reason to require redundant dictation
in the input file.  Instead, the tool can know the types and default
values by parsing the .config file.

This commit changes the tool to use the CONFIG names, but ignore the
types and default values given by the input file.

This commit also fixes one bug.  Prior to this commit, it could not
move an integer-typed CONFIG with value 1.

For example, assume we are moving CONFIG_CONS_INDEX.  Please note
this is an integer type option.

Many board headers define this CONFIG as 1.

  #define CONFIG_CONS_INDEX    1

It will be converted to

  CONFIG_CONS_INDEX=y

and moved to include/autoconf.mk, by the tools/scripts/define2mk.sed.
It will cause "make savedefconfig" to fail due to the type conflict.

This commit takes care of it by detecting the type and converting the
CONFIG value correctly.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

Currently, the progress " * defconfigs out of 1133" does not increase
monotonically.

Moreover, the number of processed defconfigs does not match the total
number of defconfigs when this tool finishes, like:

 1132 defconfigs out of 1133
Clean up headers? [y/n]:

It looks like the task was not completed, and some users might feel
upset about it.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

When the source tree is not clean, this tool raises an exception
with a message like follows:

Traceback (most recent call last):
  File "tools/moveconfig.py", line 939, in <module>
    main()
  File "tools/moveconfig.py", line 934, in main
    move_config(config_attrs, options)
  File "tools/moveconfig.py", line 808, in move_config
    while not slots.available():
  File "tools/moveconfig.py", line 733, in available
    if slot.poll():
  File "tools/moveconfig.py", line 645, in poll
    self.parser.update_dotconfig(self.defconfig)
  File "tools/moveconfig.py", line 503, in update_dotconfig
    with open(autoconf_path) as f:
IOError: [Errno 2] No such file or directory: '/tmp/tmpDtzCgl/include/autoconf.mk'

This does not explain what is wrong.  Show an appropriate error
message "source tree is not clean, please run 'make mrproper'"
in such a situation.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

Since commit 25400090 ("moveconfig: Print a message for
missing compiler"), this tool parses an error message every time an
error occurs during the process in order to detect missing compiler.

Instead of that, we can look for compilers in the PATH environment
only once before starting the defconfig walk.  If a desired compiler
is missing, "make include/config/auto.conf" will apparently fail for
that architecture.  So, the tool can just skip those board, showing
"Compiler is missing.  Do nothing.".
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

We must ensure this tool is run from the top of source directory
before calling update_cross_compile().  Otherwise, the following
exception is thrown:

Traceback (most recent call last):
  File "./moveconfig.py", line 918, in <module>
    main()
  File "./moveconfig.py", line 908, in main
    update_cross_compile()
  File "./moveconfig.py", line 292, in update_cross_compile
    for arch in os.listdir('arch'):
OSError: [Errno 2] No such file or directory: 'arch'
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

Nesting by "else:" is not generally useful after such statements
as return, break, sys.exit(), etc.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

Commit 96464bad ("moveconfig: Always run savedefconfig on the
moved config") changed how defconfig files were updated.

Since then, the function update_defconfig() does not modify defconfig
files at all (instead, they are updated by "make savedefconfig"), so
update_dotconfig() is a better fit for this function.  Also, update
the comment block to match the actual behavior.
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

Since commit 96464bad ("moveconfig: Always run savedefconfig on
the moved config"), --dry-run option is broken.

The --dry-run option prevents the .config from being modified,
but defconfig files might be updated by "make savedefconfig"
regardless of the --dry-run option.

Move the "if not self.options.dry_run" conditional to the correct
place.

Fixes 96464bad ("moveconfig: Always run savedefconfig on the moved config")
Signed-off-by: NMasahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: NJoe Hershberger <joe.hershberger@ni.com>

Spelling corrections for (among other things):

* environment
* override
* variable
* ftd (should be "fdt", for flattened device tree)
* embedded
* FTDI
* emulation
* controller

Since commit 87da2690
"openrisc: updating build tools naming convention", openrisc
kernel.org toolchain is out of date and cannot build U-Boot.
Update buildman and moveconfig tools to refer to the new one.
Signed-off-by: NBin Meng <bmeng.cn@gmail.com>

Give a full URL for a working nds32 toolchain for U-Boot.
Signed-off-by: NBin Meng <bmeng.cn@gmail.com>
Reviewed-by: NMasahiro Yamada <yamada.masahiro@socionext.com>