Changed the way unicharsets are handled to allow support for the ™ character. Can find the issue where it was requested.

This fixes these compiler warnings:

    ccmain/equationdetect.cpp:1519:2: warning: extra ‘;’ [-Wpedantic]
    ccstruct/blobs.cpp:65:17: warning: extra ‘;’ [-Wpedantic]
    ccstruct/blobs.h:178:18: warning: extra ‘;’ [-Wpedantic]
    ccstruct/ratngs.cpp:36:22: warning: extra ‘;’ [-Wpedantic]
    ccstruct/ratngs.cpp:37:22: warning: extra ‘;’ [-Wpedantic]
    ccutil/ambigs.cpp:46:20: warning: extra ‘;’ [-Wpedantic]
    ccutil/ambigs.h:137:21: warning: extra ‘;’ [-Wpedantic]
    cutil/structures.cpp:36:45: warning: extra ‘;’ [-Wpedantic]
    textord/equationdetectbase.cpp:65:2: warning: extra ‘;’ [-Wpedantic]
    textord/equationdetectbase.h:57:2: warning: extra ‘;’ [-Wpedantic]
    wordrec/lm_state.cpp:25:28: warning: extra ‘;’ [-Wpedantic]
    wordrec/lm_state.h:190:29: warning: extra ‘;’ [-Wpedantic]
Signed-off-by: NStefan Weil <sw@weilnetz.de>

Compiler warning:

    ccutil/unicharcompress.cpp:76:76: warning: format ‘%x’ expects argument of type ‘unsigned int*’, but argument 3 has type ‘int*’ [-Wformat=]
    ccutil/unicharcompress.cpp:80:31: warning: format ‘%x’ expects argument of type ‘unsigned int*’, but argument 3 has type ‘int*’ [-Wformat=]
Signed-off-by: NStefan Weil <sw@weilnetz.de>

Move LSTM unicharset and recoder to traineddata with version string part1. Backwards compatible - maybe.

Added script-specific validation and normalization for virama-using scripts and updated normalization for others

Fixes from pull of cleanups: clang tidied, reviewed, fixed new bugs, undeleted needed code. Probably breaks the build, due to some inclusion of changes in utf8/32 conversion

MS Visual Studio does not provide that function, but can use strtok_s
which does exactly the same.
Signed-off-by: NStefan Weil <sw@weilnetz.de>

The member variable default_sid_ was used without being initialized.

Valgrind report for `tesseract --oem 1 hello.png hello`:

    Conditional jump or move depends on uninitialised value(s)
       at 0x14352E: BITS16::set_bit(unsigned char, unsigned char) (bits16.h:50)
       by 0x143E27: WERD::set_flag(WERD_FLAGS, unsigned char) (werd.h:129)
       by 0x27D053: WERD_RES::SetupWordScript(UNICHARSET const&) (pageres.cpp:381)
       by 0x27CAFD: WERD_RES::SetupForRecognition(UNICHARSET const&, tesseract::Tesseract*, Pix*, int, TBOX const*, bool, bool, bool, ROW*, BLOCK const*) (pageres.cpp:316)
       by 0x145903: tesseract::Tesseract::SetupWordPassN(int, tesseract::WordData*) (control.cpp:182)
       by 0x145780: tesseract::Tesseract::SetupAllWordsPassN(int, TBOX const*, char const*, PAGE_RES*, GenericVector<tesseract::WordData>*) (control.cpp:168)
       by 0x146293: tesseract::Tesseract::recog_all_words(PAGE_RES*, ETEXT_DESC*, TBOX const*, char const*, int) (control.cpp:336)
       by 0x12F356: tesseract::TessBaseAPI::Recognize(ETEXT_DESC*) (baseapi.cpp:878)
       by 0x13036D: tesseract::TessBaseAPI::ProcessPage(Pix*, int, char const*, char const*, int, tesseract::TessResultRenderer*) (baseapi.cpp:1184)
       by 0x13014A: tesseract::TessBaseAPI::ProcessPagesInternal(char const*, char const*, int, tesseract::TessResultRenderer*) (baseapi.cpp:1140)
       by 0x12FBCE: tesseract::TessBaseAPI::ProcessPages(char const*, char const*, int, tesseract::TessResultRenderer*) (baseapi.cpp:1040)
       by 0x12C3DF: main (tesseractmain.cpp:515)
     Uninitialised value was created by a heap allocation
       at 0x4C2C21F: operator new(unsigned long) (vg_replace_malloc.c:334)
       by 0x12D88B: tesseract::TessBaseAPI::Init(char const*, int, char const*, tesseract::OcrEngineMode, char**, int, GenericVector<STRING> const*, GenericVector<STRING> const*, bool, bool (*)(STRING const&, GenericVector<char>*)) (baseapi.cpp:320)
       by 0x12D6DA: tesseract::TessBaseAPI::Init(char const*, char const*, tesseract::OcrEngineMode, char**, int, GenericVector<STRING> const*, GenericVector<STRING> const*, bool) (baseapi.cpp:284)
       by 0x12C088: main (tesseractmain.cpp:440)
Signed-off-by: NStefan Weil <sw@weilnetz.de>

Signed-off-by: NStefan Weil <sw@weilnetz.de>

Replacing inttypes.h by cinttypes fixes a problem with glibc < 2.18:
In older inttypes.h, the standard C format macros are only defined for
C++ when the macro __STDC_FORMAT_MACROS is set.
Signed-off-by: NStefan Weil <sw@weilnetz.de>

Signed-off-by: NStefan Weil <sw@weilnetz.de>

Several code locations call that method with a normal C string,
so overload it to accept that without a conversion to a STRING
object. This saves unneeded new / memcpy / delete operations.
Signed-off-by: NStefan Weil <sw@weilnetz.de>

Tesseract crashes with an unhandled exception (std::bad_alloc) if it gets
a bad tessdata file where the numEntries data field is very large (also
after swapping), for example 0x77777777.
Signed-off-by: NStefan Weil <sw@weilnetz.de>

Signed-off-by: NStefan Weil <sw@weilnetz.de>

Signed-off-by: NStefan Weil <sw@weilnetz.de>

It increased the script_table too early, so the last element was never
used.
Signed-off-by: NStefan Weil <sw@weilnetz.de>

There is no need to initialize memory with a fixed value which is
overwritten in the next step.
Signed-off-by: NStefan Weil <sw@weilnetz.de>

It's not necessary to initialize the vector with 0,
because the initial values are read from file.

Fix also an assertion when trying to read an empty file.
Signed-off-by: NStefan Weil <sw@weilnetz.de>

Signed-off-by: NStefan Weil <sw@weilnetz.de>

There is no need for an intermediate variable char_buffer.
Signed-off-by: NStefan Weil <sw@weilnetz.de>

Signed-off-by: NStefan Weil <sw@weilnetz.de>

There exist standard macro definitions for the printf format specifiers.
MS Visual Studio does not support that standard (at least not in older
versions), so local definitions are needed there.
Signed-off-by: NStefan Weil <sw@weilnetz.de>

The local variable intval is of type int.
Signed-off-by: NStefan Weil <sw@weilnetz.de>

Commit 907de599 tried to improve
GenericVector, but missed a case where vectors with less than
kDefaultVectorSize were allocated. This resulted in additional
alloc / free operations.

Commit a28b2a03 (before memory optimization)
oem 0: total heap usage: 739,238 allocs, 739,237 frees, 161,699,214 bytes allocated
oem 1: total heap usage: 690,182 allocs, 690,175 frees, 144,470,400 bytes allocated
oem 2: total heap usage: 728,213 allocs, 728,206 frees, 182,885,824 bytes allocated

Commit fd3f8f9b without genericvector change
oem 0: total heap usage: 738,980 allocs, 738,979 frees, 161,697,150 bytes allocated
oem 1: total heap usage: 690,182 allocs, 690,175 frees, 144,470,400 bytes allocated
oem 2: total heap usage: 728,213 allocs, 728,206 frees, 182,885,824 bytes allocated
=> Improvements for oem 0, no change for oem 1 and oem 2.

Commit fd3f8f9b
oem 0: total heap usage: 772,648 allocs, 772,647 frees, 160,083,901 bytes allocated
oem 1: total heap usage: 748,591 allocs, 748,584 frees, 143,581,672 bytes allocated
oem 2: total heap usage: 764,796 allocs, 764,789 frees, 181,212,197 bytes allocated
=> Less bytes allocated, but more allocs / frees = bad for performance.

Commit fd3f8f9b with this patch
oem 0: total heap usage: 677,537 allocs, 677,536 frees, 160,444,634 bytes allocated
oem 1: total heap usage: 653,812 allocs, 653,805 frees, 143,423,008 bytes allocated
oem 2: total heap usage: 670,029 allocs, 670,022 frees, 181,517,760 bytes allocated
=> Improvements for all three cases.
Signed-off-by: NStefan Weil <sw@weilnetz.de>

This fixes some compiler warnings.
Signed-off-by: NStefan Weil <sw@weilnetz.de>

Both functions simply call malloc, free.

Remove also unneeded null pointer checks and use calloc where possible.
Signed-off-by: NStefan Weil <sw@weilnetz.de>

Added extra Init that takes a memory buffer or a filereader function pointer to enable read of traineddata from memory or foreign file systems. Updated existing readers to use TFile API instead of FILE. This does not yet add big-endian capability to LSTM, but it is very easy from here.

Signed-off-by: NStefan Weil <sw@weilnetz.de>

This fixes a compiler warning:

ccutil/scanutils.cpp:284:7: warning:
 variable 'sign' set but not used [-Wunused-but-set-variable]
Signed-off-by: NStefan Weil <sw@weilnetz.de>

The code in ccutil/hashfn.h was needed for some old compilers. Now that we support MSVC >= 2010 and compilers that has good support for C++11, we can drop this code.

As a result of this file removal, we now use:
  std::unordered_map
  std::unordered_set
  std::unique_ptr
directly in the codebase with '#include' for the needed headers.