This patch adds a possibility to dynamically switch UBI self-checks
on and off, instead of toggling them compile-time from the configuration
menu. This is much more flexible, and consistent with UBIFS, and this
also simplifies UBI Kconfig menu and the code.

This patch introduces two levels of self-checks - general, which
includes all self-checks which are relatively fast, and I/O, which
includes write-verify checks and erase-verify checks, which are
relatively slow and involve flash I/O.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

This additional little macro is used to print a bit more messages
while scanning the media. However, we have the 'dbg_bld()' macro
for this, so we better us 'dbg_bld()' and kill UBI_IO_DEBUG. This
simplifies the code a tiny bit.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

Instead of using pre-allocated 'ubi->dbg_peb_buf' buffer in
'ubi_dbg_check_all_ff()', dynamically allocate it when needed. The
intend is to get rid of the pre-allocated 'ubi->dbg_peb_buf' buffer
completely. And the need for this arises because we want to change
to dynamic debugging control instead of compile-time control, i.e.,
we are going to kill the CONFIG_MTD_UBI_DEBUG_PARANOID Kconfig
option, which would mean that 'ubi->dbg_peb_buf' is always allocated,
which would be wasteful.

Thus, we are getting rid of 'ubi->dbg_peb_buf', and this is a
preparation for that.
signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

Instead of using pre-allocated 'ubi->dbg_peb_buf' buffer in
'ubi_dbg_check_write()', dynamically allocate it when needed. The
intend is to get rid of the pre-allocated 'ubi->dbg_peb_buf' buffer
completely. And the need for this arises because we want to change
to dynamic debugging control instead of compile-time control, i.e.,
we are going to kill the CONFIG_MTD_UBI_DEBUG_PARANOID Kconfig
option, which would mean that 'ubi->dbg_peb_buf' is always allocated,
which would be wasteful.

Thus, we are getting rid of 'ubi->dbg_peb_buf', and this is a
preparation for that.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

When the read operation fails, UBI tries to re-read several times in
a hope that one of the subsequent reads may succeed. However, currently
UBI re-reads only if MTD failed to read all data, but does not re-reads
if all the data were read, but with an integrity error (-EBADMSB). This
patch makes UBI to always re-try reading.

This should be useful for reading NAND pages with unstable bits -
re-reading may help to get correct data.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

This change affects only the debugging code. Namely, use mtd->read()
function instead of ubi_io_read() to avoid bit-flips injection
(ubi_dbg_is_bitflip()) which we do not want on the debugging path.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

When reading data from the flash, corrupt the buffer we are about to
read to. The idea is to fix the following possible situation:

1. The buffer contains data from previous operation, e.g., read from
   another PEB previously. The data looks like expected, e.g., if we
   just do not read anything and return - the caller would not
   notice this. E.g., if we are reading a VID header, the buffer may
   contain a valid VID header from another PEB.
2. The driver is buggy and returns use success or -EBADMSG or
   -EUCLEAN, but it does not actually put any data to the buffer.

This may confuse UBI or upper layers - they may think the buffer
contains valid data while in fact it is just old data.

Thus, try to reveal such buggy MTD drivers with simple debugging
code which fills the read buffer with 0x12 constant.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

In 'nor_erase_prepare()' we want to make sure the UBI headers are
corrupted. But it is possible that one of the headers just contains
all 0xFFs, which is also OK, because UBI will erase it in case of
a power cut.
Signed-off-by: NHolger Brunck <holger.brunck@keymile.com>
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

My new shiny code for corrupted PEB detection has NOR specific bug.
We tread PEB as corrupted and preserve it, if

1. EC header is OK.
2. VID header is corrupted.
3. data area is not "all 0xFFs"

In case of NOR we have 'nor_erase_prepare()' quirk, which invalidates
the headers before erasing the PEB. And we invalidate first the VID
header, and then the EC header. So if a power cut happens after we have
invalidated the VID header, but before we have invalidated the EC
header, we end up with a PEB which satisfies the above 3 conditions,
and the scanning code will treat it as corrupted, and will print
scary warnings, wrongly.

This patch fixes the issue by firt invalidating the EC header, then
invalidating the VID header. In case of power cut inbetween, we still
just lose the EC header, and UBI can deal with this situation gracefully.

Thanks to Anatolij Gustschin <agust@denx.de> for tracking this down.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>
Reported-by: NAnatolij Gustschin <agust@denx.de>
Tested-by: NAnatolij Gustschin <agust@denx.de>

This patch turns static function 'check_pattern()' into a non-static
'ubi_check_pattern()'. This is just a preparation for the chages which
are coming in the next patches.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

Currently UBI has one small flaw - when we read EC or VID header, but find only
0xFF bytes, we return UBI_IO_FF and do not report whether we had bit-flips or
not. In case of the VID header, the scanning code adds this PEB to the free list,
even though there were bit-flips.

Imagine the following situation: we start writing VID header to a PEB and have a
power cut, so the PEB becomes unstable. When we scan and read the PEB, we get
a bit-flip. Currently, UBI would just ignore this and treat the PEB as free. This
patch changes UBI behavior and now UBI will schedule this PEB for erasure.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

The 'UBI_IO_PEB_EMPTY' and 'UBI_IO_PEB_FREE' are essentially the same
and mean that there are only 0xFF bytes instead of headers. Simplify
UBI a little by turning them into a single 'UBI_IO_FF' error code.

Also, stop maintaining commentaries in 'ubi_io_read_vid_hdr()' which are
almost identical to commentaries in 'ubi_io_read_ec_hdr()'.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

Rename UBI_IO_BAD_HDR_READ into UBI_IO_BAD_HDR_EBADMSG which is presumably more
self-documenting and readable. Indeed, the '_READ' suffix does not tell much and
even confuses, while '_EBADMSG' tells about uncorrectable ECC error, because we
use -EBADMSG all over the place to represent ECC errors.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

Fix the followong compilation warnings introduced by commit
095751a6:

drivers/mtd/ubi/scan.c: In function 'check_what_we_have':
drivers/mtd/ubi/scan.c:960: warning: passing argument 1 of 'get_random_bytes' discards qualifiers from pointer target type

Fix the following compilation warnings introduced by commit
1a49af2c:

drivers/mtd/ubi/io.c: In function 'ubi_io_read':
drivers/mtd/ubi/io.c:153: warning: initialization makes integer from pointer without a cast
drivers/mtd/ubi/io.c:170: warning: format '%s' expects type 'char *', but argument 5 has type 'int'
drivers/mtd/ubi/io.c:177: warning: format '%zd' expects type 'signed size_t', but argument 7 has type 'int'
drivers/mtd/ubi/io.c:177: warning: too many arguments for format

Also, amend the ECC error code string and add brackets and whitespace
there - this should make the message readable.
Reported-by: NStephen Rothwell <sfr@canb.auug.org.au>
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

ECC errors are quite typical errors on NAND, so it is worth improving
the UBI message and print something like

ubi_io_read: error -74 (ECC error) while reading 4096 bytes from PEB 1:4 ...

rather than

ubi_io_read: error -74 while reading 4096 bytes from PEB 1:4 ...
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

This patch introduces the %UBI_IO_BAD_HDR_READ return code for
the I/O level function. We will use this code in order to distinguish
between "corrupted header possibly because this is non-ubi data" and
"corrupted header possibly because of real data corruption and ECC error".

So far this patch does not introduce any functional change, just a
preparation.

This patch is pased on a patch from
Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>
Reviewed-by: NSebastian Andrzej Siewior <sebastian@breakpoint.cc>
Tested-by: NSebastian Andrzej Siewior <sebastian@breakpoint.cc>

We do not really need 2 separate error codes for indicating bad VID
and bad EC headers (UBI_IO_BAD_EC_HDR, UBI_IO_BAD_VID_HDR), it is
enough to have only one UBI_IO_BAD_HDR return code.

This patch does not introduce any functional change, only some
code simplification.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>
Reviewed-by: NSebastian Andrzej Siewior <sebastian@breakpoint.cc>
Tested-by: NSebastian Andrzej Siewior <sebastian@breakpoint.cc>

Signed-off-by: NShinya Kuribayashi <shinya.kuribayashi.px@renesas.com>
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

Signed-off-by: NShinya Kuribayashi <shinya.kuribayashi.px@renesas.com>
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h

percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files.  percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.

percpu.h -> slab.h dependency is about to be removed.  Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability.  As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.

  http://userweb.kernel.org/~tj/misc/slabh-sweep.py

The script does the followings.

* Scan files for gfp and slab usages and update includes such that
  only the necessary includes are there.  ie. if only gfp is used,
  gfp.h, if slab is used, slab.h.

* When the script inserts a new include, it looks at the include
  blocks and try to put the new include such that its order conforms
  to its surrounding.  It's put in the include block which contains
  core kernel includes, in the same order that the rest are ordered -
  alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
  doesn't seem to be any matching order.

* If the script can't find a place to put a new include (mostly
  because the file doesn't have fitting include block), it prints out
  an error message indicating which .h file needs to be added to the
  file.

The conversion was done in the following steps.

1. The initial automatic conversion of all .c files updated slightly
   over 4000 files, deleting around 700 includes and adding ~480 gfp.h
   and ~3000 slab.h inclusions.  The script emitted errors for ~400
   files.

2. Each error was manually checked.  Some didn't need the inclusion,
   some needed manual addition while adding it to implementation .h or
   embedding .c file was more appropriate for others.  This step added
   inclusions to around 150 files.

3. The script was run again and the output was compared to the edits
   from #2 to make sure no file was left behind.

4. Several build tests were done and a couple of problems were fixed.
   e.g. lib/decompress_*.c used malloc/free() wrappers around slab
   APIs requiring slab.h to be added manually.

5. The script was run on all .h files but without automatically
   editing them as sprinkling gfp.h and slab.h inclusions around .h
   files could easily lead to inclusion dependency hell.  Most gfp.h
   inclusion directives were ignored as stuff from gfp.h was usually
   wildly available and often used in preprocessor macros.  Each
   slab.h inclusion directive was examined and added manually as
   necessary.

6. percpu.h was updated not to include slab.h.

7. Build test were done on the following configurations and failures
   were fixed.  CONFIG_GCOV_KERNEL was turned off for all tests (as my
   distributed build env didn't work with gcov compiles) and a few
   more options had to be turned off depending on archs to make things
   build (like ipr on powerpc/64 which failed due to missing writeq).

   * x86 and x86_64 UP and SMP allmodconfig and a custom test config.
   * powerpc and powerpc64 SMP allmodconfig
   * sparc and sparc64 SMP allmodconfig
   * ia64 SMP allmodconfig
   * s390 SMP allmodconfig
   * alpha SMP allmodconfig
   * um on x86_64 SMP allmodconfig

8. percpu.h modifications were reverted so that it could be applied as
   a separate patch and serve as bisection point.

Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: NTejun Heo <tj@kernel.org>
Guess-its-ok-by: NChristoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>

Add an extra debugging check function which validates writes.
After every write it reads the data back, compares it with the
original data, and complains if they mismatch.

Useful for debugging. No-op if extra debugging checks are disabled.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

UBI debugging functions were a little bit over-engineered and
returned more error codes than needed, and the callers had to
do useless checks. Simplify the return codes.

Impact: only debugging code is affected, which means that for
        non-developers this is a no-op patch.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

More testing of NOR flash against power cuts showed that sometimes
eraseblocks may be unwritable, and we cannot really invalidate
them before erasure. But in this case the eraseblock probably
contains garbage anyway, and we do not have to invalidate the
headers. This assumption might be not true, but this is at least
what I have observed. So if we cannot invalidate the headers,
we make sure that the PEB does not contain valid VID header.
If this is true, everything is fine, otherwise we panic.

Useful for debugging problems, compiled in only if UBI debugging
is enabled. This patch also makes the UBI writing function dump
the flash if it fails to write.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

In case of NOR flash, UBI zeroes EC and VID headers' magic,
in order to detect interrupted erasures. It first zeroes out
the EC magic, then VID magic. However, if a power cut happens
in between, we'll end up with a corrupted EC header and a valid
VID header, in which case UBI accepts the PEB, but prints a
warning. This patch makes sure we first zero out the VID
magic, then the EC magic, not vice versa. This is just a
small amendment to prevent warning messages.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

The recent "UBI: fix NOR flash recovery" introduced compilation
warnings which were immediately spotted by our linux-next keeper.
This patch fixes them.
Reported-by: NStephen Rothwell <sfr@canb.auug.org.au>
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

This commit fixes NOR flash recovery issues observed with Spansion
S29GL512N NOR.

When NOR erases, it first fills PEBs with zeroes, then sets all bytes
to 0xFF. Filling with zeroes starts from the end of the PEB. And when
power is cut, this results in PEBs containing correct EC and VID headers
but corrupted with zeros at the end. This confuses UBI and it mistakinly
accepts these PEBs and associate them with LEBs.

Fis this issue by zeroing EC and VID magics before erasing PEBs, to
make UBI later refuse zem.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

Move the image seq. number handling from I/O level to the scanning
lever, where it really belongs to. Move the @image_seq_set variable
to the @struct ubi_scan_info structure, which exists only during
scanning.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

An image sequence number is added to the UBI erase-counter header
to be able determine if the root file system contains a mixture
of old and new images (because the flashing failed to complete).

A change to nolo is also needed for this to take effect.
Signed-off-by: NAdrian Hunter <adrian.hunter@nokia.com>
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

The 'paranoid_check_empty()' is bogus because, which is easilly
seen on NOR flash, which has long erase cycles, and which may
easilly end-up with half-erased eraseblocks. In this case the
paranoid check fails. I is just wrong to assume that PEBs which
do not have EC headers always contain all 0xFF. Such assumption
should not be made on the I/O level, which is quite low.

Thus, just kill the check.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

This patch adds code which makes sure eraseblocks contain all 0xFF
bytes before starting using them. The verification is done only when
debugging checks are enabled.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

Some of the typos were indicated by Adrian Hunter,
some by 'aspell'.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

When paranoid checs are enabled, the 'io_paral' test from the
'mtd-utils' package fails. The symptoms are:

UBI error: paranoid_check_all_ff: flash region at PEB 3973:512, length 15872 does not contain all 0xFF bytes
UBI error: paranoid_check_all_ff: paranoid check failed for PEB 3973
UBI: hex dump of the 512-16384 region

It turned out to be a bug in the checking function. Suppose there
are 2 tasks - A and B. Task A is the wear-levelling working
('wear_leveling_worker()'). It is reading the VID header to find
which LEB this PEB belongs to. Say, task A is reading header
of PEB X. Suppose PEB X is unmapped, and has no VID header.
Task B is trying to write to PEB X.

Task A: in 'ubi_io_read_vid_hdr()': reads the VID header from PEB X.
        The read data contain all 0xFF bytes.
Task B: writes VID header and some data to PEB X
Task A: assumes PEB X is empty, calls 'paranoid_check_all_ff()', which
        fails.

The solution for this problem is to make 'paranoid_check_all_ff()'
re-read the VID header, re-check it, and only if it is not there,
check the rest. This now implemented by the 'paranoid_check_empty()'
function.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

- (better, more, bigger ...) then -> (...) than
Signed-off-by: NFrederik Schwarzer <schwarzerf@gmail.com>
Signed-off-by: NJiri Kosina <jkosina@suse.cz>

Just minor indentation and "over 80 characters" fixes.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

The 'ubi_io_read_vid_hdr()' and 'ubi_io_read_ec_hdr()' function
have the 'verbose' argument which controls whether they should
print a warning if the VID/EC header was not found or was corrupted.
Some callers require the headers to be OK, and pass 1. Some allow
a corrupted/not present header, and pass 0.

       if (UBI_IO_DEBUG)
               verbose = 1;

And UBI_IO_DEBUG is 1 if CONFIG_MTD_UBI_DEBUG_MSG_BLD is true. So in
this case the warning is printed all the time. This confuses people.

Thus, do not print the messages as warnings if UBI_IO_DEBUG is true,
but print them as debugging messages instead.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

No functional changes, just tweak comments to make kernel-doc
work fine and stop complaining.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

Just out or curiousity ran checkpatch.pl for whole UBI,
and discovered there are quite a few of stylistic issues.
Fix them.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>

If bit-flips happen often, UBI prints to many messages. Lessen
the amount by only printing the messages when the PEB has been
scrubbed. Also, print torturing messages.
Signed-off-by: NArtem Bityutskiy <Artem.Bityutskiy@nokia.com>