At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).
Signed-off-by: NSimon Glass <sjg@chromium.org>

At present binman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move binman to use absolute imports. This enables removable of the path
adjusting in Entry also.
Signed-off-by: NSimon Glass <sjg@chromium.org>

This code is useful in other tools. Move it into a common file so it can
be shared.
Signed-off-by: NSimon Glass <sjg@chromium.org>

We don't need to run test coverage with Python 2 now. Drop the
special-case code.
Signed-off-by: NSimon Glass <sjg@chromium.org>

Python does not like the module name being the same as the module
directory. To allow buildman modules to be used from other tools, rename
it.
Signed-off-by: NSimon Glass <sjg@chromium.org>

We can rely on Python 3 now, so drop the workaround for importing
StringIO.
Signed-off-by: NSimon Glass <sjg@chromium.org>

We don't need these now that the tools using Python 3. Drop them.
Signed-off-by: NSimon Glass <sjg@chromium.org>

Python does not like the module name being the same as the module
directory. To allow patman modules to be used from other tools, rename
it.
Signed-off-by: NSimon Glass <sjg@chromium.org>

When outputing a progress line we don't want it to go past the end of the
current terminal line, or it will not be possible to erase it. Add an
option to Print() which allows limiting the output to the terminal width.

Since ANSI sequences do not take up space on the terminal, these are
removed.
Signed-off-by: NSimon Glass <sjg@chromium.org>

When printing progress it is useful to print a message and leave the
cursor at the end of the line until the operation is finished. When it is
finished, the line needs to be erased so a new line can start in its place.

Add a function to handle clearing a line previously written by
terminal.Print()
Signed-off-by: NSimon Glass <sjg@chromium.org>

This does not seem to work on Python 3. Update the code to use the
built-in support.
Signed-off-by: NSimon Glass <sjg@chromium.org>

Quite often on a series that has clean-up patches, the individual patches
may fit within the cc limit but the cover letter does not. Apply the same
limit to the cover letter.
Signed-off-by: NSimon Glass <sjg@chromium.org>
Reviewed-by: NChris Packham <judge.packham@gmail.com>

fixes: 8ab452d5

When compiling list of cover letter cc addresses, using null as a
separater, then encoding to utf-8 results in lots of "\x00" as
separators. patman then doesnt understand that when it comes to
repoting the list to send-email.

Fix this by not encoding to utf-8, as done for the other patch files.
Signed-off-by: NRobert Beckett <bob.beckett@collabora.com>
Reviewed-by: NSimon Glass <sjg@chromium.org>

s/Subprocress/Subprocess/
s/easiler/easier/
s/repositiory/repository/
s/rangem/range/
s/Retruns/Returns/
Signed-off-by: NAnatolij Gustschin <agust@denx.de>
Reviewed-by: NSimon Glass <sjg@chromium.org>

Update this tool to use Python 3 to meet the 2020 deadline.
Signed-off-by: NSimon Glass <sjg@chromium.org>

At present patman test fail in some environments which don't use utf-8
as the default file encoding. Add this explicitly.
Signed-off-by: NSimon Glass <sjg@chromium.org>

At present all the 'command' methods return bytes. Most of the time we
actually want strings, so change this. We still need to keep the internal
representation as bytes since otherwise unicode strings might break over
a read() boundary (e.g. 4KB), causing errors. But we can convert the end
result to strings.

Add a 'binary' parameter to cover the few cases where bytes are needed.
Signed-off-by: NSimon Glass <sjg@chromium.org>

There is a contributor in Linux kernel with a comma in their name, which
confuses patman and results in invalid to- or cc- addresses on some
patches. To avoid this, let's use \0 as a separator when generating cc
file.
Signed-off-by: NDmitry Torokhov <dmitry.torokhov@gmail.com>
Reviewed-by: NSimon Glass <sjg@chromium.org>

As per the centithread on ksummit-discuss [1], there are folks who
feel that if a Change-Id is present in a developer's local commit that
said Change-Id could be interesting to include in upstream posts.
Specifically if two commits are posted with the same Change-Id there's
a reasonable chance that they are either the same commit or a newer
version of the same commit.  Specifically this is because that's how
gerrit has trained people to work.

There is much angst about Change-Id in upstream Linux, but one thing
that seems safe and non-controversial is to include the Change-Id as
part of the string of crud that makes up a Message-Id.

Let's give that a try.

In theory (if there is enough adoption) this could help a tool more
reliably find various versions of a commit.  This actually might work
pretty well for U-Boot where (I believe) quite a number of developers
use patman, so there could be critical mass (assuming that enough of
these people also use a git hook that adds Change-Id to their
commits).  I was able to find this git hook by searching for "gerrit
change id git hook" in my favorite search engine.

In theory one could imagine something like this could be integrated
into other tools, possibly even git-send-email.  Getting it into
patman seems like a sane first step, though.

NOTE: this patch is being posted using a patman containing this patch,
so you should be able to see the Message-Id of this patch and see that
it contains my local Change-Id, which ends in 2b9 if you want to
check.

[1] https://lists.linuxfoundation.org/pipermail/ksummit-discuss/2019-August/006739.htmlSigned-off-by: NDouglas Anderson <dianders@chromium.org>

Remove this file from git and instead build it using the Makefile.

Update tools.GetInputFilename() to support reading files from an absolute
path, so that we can read the Elf test files easily. Also make sure that
the temp directory is report in ELF tests as this was commented out.
Signed-off-by: NSimon Glass <sjg@chromium.org>

At present tools are not expected to fail. If they do an exception is
raised but there is no detail about what went wrong. This makes it hard
to debug if something does actually go wrong.

Fix this by outputting both stderr and stdout on failure.
Signed-off-by: NSimon Glass <sjg@chromium.org>

Since cros_subprocess use bytestrings now, this feature not needed. Drop
it.
Signed-off-by: NSimon Glass <sjg@chromium.org>

At present outdir remains set ever after the output directory has been
removed. Fix this to avoid trying to access it when it is not present.
Signed-off-by: NSimon Glass <sjg@chromium.org>

At present binman cannot replace data within a CBFS since it does not
allow rewriting of the files in that CBFS. Implement this by using the
new WriteData() method to handle the case.

Add a header to compressed data so that the amount of compressed data can
be determined without reference to the size of the containing entry. This
allows the entry to be larger that the contents, without causing errors in
decompression. This is necessary to cope with a compressed device tree
being updated in such a way that it shrinks after the entry size is
already set (an obscure case). It is not used with CBFS since it has its
own metadata for this. Increase the number of passes allowed to resolve
the position of entries, to handle this case.

Add a test for this new logic.
Signed-off-by: NSimon Glass <sjg@chromium.org>

Use the new logging feature to log information about progress with
packing. This is useful to see how binman is figuring things out.

Also update elf.py to use the same feature.
Signed-off-by: NSimon Glass <sjg@chromium.org>

Use the debug level constants instead of open-coding them in the file.
Signed-off-by: NSimon Glass <sjg@chromium.org>

At present tout writes directly to stdout. This is not necessary and it
prevents tests from redirecting output. Change it to use print() for the
non-progress output.
Signed-off-by: NSimon Glass <sjg@chromium.org>

Binman generally operates silently but in some cases it is useful to see
what Binman is actually doing at each step. Enable some logging output
with different logging levels selectable via the -v flag.
Signed-off-by: NSimon Glass <sjg@chromium.org>

At present entry modules can only be accessed using Entry.Lookup() or
Entry.Create(). Most of the time this is fine, but sometimes a module
needs to provide constants or helper functions useful to other modules.
It is easier in this case to use 'import'.

Add an __init__ file to permit this.
Signed-off-by: NSimon Glass <sjg@chromium.org>

This class is the new way to handle arguments in Python. Convert binman
over to use it. At the same time, introduce commands so that we can
separate out the different parts of binman functionality.
Signed-off-by: NSimon Glass <sjg@chromium.org>

This tool has quite a few arguments and options, so put the functionality
in a function so that we call it from one place and hopefully get it
right.
Signed-off-by: NSimon Glass <sjg@chromium.org>

Add utility functions to compress and decompress using lz4 and lzma
algorithms. In the latter case these use the legacy lzma support favoured
by coreboot's CBFS.

No tests are provided as these functions will be tested by the CBFS
tests in a separate patch.
Signed-off-by: NSimon Glass <sjg@chromium.org>

Sometimes tools can be located by looking in other locations. Add a way to
direct the search.
Signed-off-by: NSimon Glass <sjg@chromium.org>

If kwargs contains raise_on_error then this function generates an error
due to a duplicate argument. Fix this.
Signed-off-by: NSimon Glass <sjg@chromium.org>

We need slightly different commands to run code coverage with Python 3.
Update the RunTestCoverage() function to handle this.
Signed-off-by: NSimon Glass <sjg@chromium.org>

While reading files in binary mode is the norm, sometimes we want to use
text mode. Add an optional parameter to handle this.
Signed-off-by: NSimon Glass <sjg@chromium.org>

In Python 3 bytes and str are separate types. Use bytes to ensure that
the code functions correctly with Python 3.
Signed-off-by: NSimon Glass <sjg@chromium.org>

The difference between the bytes and str types in Python 3 requires a
number of minor changes to this function. Update it to handle the input
data using the 'bytes' type. Create two useful helper functions which can
be used by other modules too.
Signed-off-by: NSimon Glass <sjg@chromium.org>

Update the shebang to allow either Python 2 or Python 3.
Signed-off-by: NSimon Glass <sjg@chromium.org>

Use the new functions in the tools module to handle conversion.
Signed-off-by: NSimon Glass <sjg@chromium.org>