UTF supports lossless conversion round tripping and conversions between
UTF and other encodings are mostly round trip safe as Unicode aims to be
a superset of all other character encodings. However, certain encodings
(e.g. SHIFT-JIS) are known to have round trip issues [1].

Add 'core.checkRoundtripEncoding', which contains a comma separated
list of encodings, to define for what encodings Git should check the
conversion round trip if they are used in the 'working-tree-encoding'
attribute.

Set SHIFT-JIS as default value for 'core.checkRoundtripEncoding'.

[1] https://support.microsoft.com/en-us/help/170559/prb-conversion-problem-between-shift-jis-and-unicodeSigned-off-by: NLars Schneider <larsxschneider@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Add the GIT_TRACE_WORKING_TREE_ENCODING environment variable to enable
tracing for content that is reencoded with the 'working-tree-encoding'
attribute. This is useful to debug encoding issues.
Signed-off-by: NLars Schneider <larsxschneider@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Check that new content is valid with respect to the user defined
'working-tree-encoding' attribute.
Signed-off-by: NLars Schneider <larsxschneider@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Git recognizes files encoded with ASCII or one of its supersets (e.g.
UTF-8 or ISO-8859-1) as text files. All other encodings are usually
interpreted as binary and consequently built-in Git text processing
tools (e.g. 'git diff') as well as most Git web front ends do not
visualize the content.

Add an attribute to tell Git what encoding the user has defined for a
given file. If the content is added to the index, then Git reencodes
the content to a canonical UTF-8 representation. On checkout Git will
reverse this operation.
Signed-off-by: NLars Schneider <larsxschneider@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

If the endianness is not defined in the encoding name, then let's
be strict and require a BOM to avoid any encoding confusion. The
is_missing_required_utf_bom() function returns true if a required BOM
is missing.

The Unicode standard instructs to assume big-endian if there in no BOM
for UTF-16/32 [1][2]. However, the W3C/WHATWG encoding standard used
in HTML5 recommends to assume little-endian to "deal with deployed
content" [3]. Strictly requiring a BOM seems to be the safest option
for content in Git.

This function is used in a subsequent commit.

[1] http://unicode.org/faq/utf_bom.html#gen6
[2] http://www.unicode.org/versions/Unicode10.0.0/ch03.pdf
     Section 3.10, D98, page 132
[3] https://encoding.spec.whatwg.org/#utf-16leSigned-off-by: NLars Schneider <larsxschneider@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Whenever a data stream is declared to be UTF-16BE, UTF-16LE, UTF-32BE
or UTF-32LE a BOM must not be used [1]. The function returns true if
this is the case.

This function is used in a subsequent commit.

[1] http://unicode.org/faq/utf_bom.html#bom10Signed-off-by: NLars Schneider <larsxschneider@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The function same_encoding() could only recognize alternative names for
UTF-8 encodings. Teach it to recognize all kinds of alternative UTF
encoding names (e.g. utf16).

While we are at it, fix a crash that would occur if same_encoding() was
called with a NULL argument and a non-NULL argument.

This function is used in a subsequent commit.
Signed-off-by: NLars Schneider <larsxschneider@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Check in a case insensitive manner if one string is a prefix of another
string.

This function is used in a subsequent commit.
Signed-off-by: NLars Schneider <larsxschneider@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Create a copy of an existing string and make all characters upper case.
Similar xstrdup_tolower().

This function is used in a subsequent commit.
Signed-off-by: NLars Schneider <larsxschneider@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Since 3733e694 (use xmallocz to avoid size arithmetic, 2016-02-22) we
allocate the buffer for the lower case string with xmallocz(). This
already ensures a NUL at the end of the allocated buffer.

Remove the unnecessary assignment.
Signed-off-by: NLars Schneider <larsxschneider@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

When calling convert_to_git(), the checksafe parameter defined what
should happen if the EOL conversion (CRLF --> LF --> CRLF) does not
roundtrip cleanly. In addition, it also defined if line endings should
be renormalized (CRLF --> LF) or kept as they are.

checksafe was an safe_crlf enum with these values:
SAFE_CRLF_FALSE:       do nothing in case of EOL roundtrip errors
SAFE_CRLF_FAIL:        die in case of EOL roundtrip errors
SAFE_CRLF_WARN:        print a warning in case of EOL roundtrip errors
SAFE_CRLF_RENORMALIZE: change CRLF to LF
SAFE_CRLF_KEEP_CRLF:   keep all line endings as they are

In some cases the integer value 0 was passed as checksafe parameter
instead of the correct enum value SAFE_CRLF_FALSE. That was no problem
because SAFE_CRLF_FALSE is defined as 0.

FALSE/FAIL/WARN are different from RENORMALIZE and KEEP_CRLF. Therefore,
an enum is not ideal. Let's use a integer bit pattern instead and rename
the parameter to conv_flags to make it more generically usable. This
allows us to extend the bit pattern in a subsequent commit.
Reported-By: NRandall S. Becker <rsbecker@nexbridge.com>
Helped-By: NLars Schneider <larsxschneider@gmail.com>
Signed-off-by: NTorsten Bögershausen <tboegi@web.de>
Signed-off-by: NLars Schneider <larsxschneider@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Signed-off-by: NJunio C Hamano <gitster@pobox.com>

"git describe" was taught to dig trees deeper to find a
<commit-ish>:<path> that refers to a given blob object.

* sb/describe-blob:
  builtin/describe.c: describe a blob
  builtin/describe.c: factor out describe_commit
  builtin/describe.c: print debug statements earlier
  builtin/describe.c: rename `oid` to avoid variable shadowing
  revision.h: introduce blob/tree walking in order of the commits
  list-objects.c: factor out traverse_trees_and_blobs
  t6120: fix typo in test name

"git merge" learned to pay attention to merge.verifySignatures
configuration variable and pretend as if '--verify-signatures'
option was given from the command line.

* hi/merge-verify-sig-config:
  t5573, t7612: clean up after unexpected success of 'pull' and 'merge'
  t: add tests for pull --verify-signatures
  merge: add config option for verifySignatures

Git has been taught to support an https:// URL used for http.proxy
when using recent versions of libcurl.

* ws/curl-http-proxy-over-https:
  http: support CURLPROXY_HTTPS

Doc update.

* ks/doc-previous-checkout:
  Doc/check-ref-format: clarify information about @{-N} syntax

Error messages from "git rebase" have been somewhat cleaned up.

* ks/rebase-error-messages:
  rebase: rebasing can also be done when HEAD is detached
  rebase: distinguish user input by quoting it
  rebase: consistently use branch_name variable

Docfix.

* sr/http-sslverify-config-doc:
  config: document default value of http.sslVerify

"git imap-send" did not correctly quote the folder name when
making a request to the server, which has been corrected.

* nm/imap-send-quote-server-folder-name:
  imap-send: URI encode server folder

Test fix.

* bp/fsmonitor:
  p7519: improve check for prerequisite WATCHMAN

* jh/partial-clone-doc:
  partial-clone: design doc

Code clean-up.

* jt/transport-hide-vtable:
  transport: make transport vtable more private
  clone, fetch: remove redundant transport check

"git version --build-options" learned to report the host CPU and
the exact commit object name the binary was built from.

* js/enhanced-version-info:
  version --build-options: report commit, too, if possible
  version --build-options: also report host CPU

* tz/lib-git-svn-svnserve-tests:
  t/lib-git-svn.sh: improve svnserve tests with parallel make test
  t/lib-git-svn: cleanup inconsistent tab/space usage

"git svn" has been updated to strip CRs in the commit messages, as
recent versions of Subversion rejects them.

* ew/svn-crlf:
  git-svn: convert CRLF to LF in commit message to SVN

Introduce a helper to simplify code to parse a common pattern that
expects either "--key" or "--key=<something>".

* cc/skip-to-optional-val:
  t4045: reindent to make helpers readable
  diff: add tests for --relative without optional prefix value
  diff: use skip_to_optional_arg_default() in parsing --relative
  diff: use skip_to_optional_arg_default()
  diff: use skip_to_optional_arg()
  index-pack: use skip_to_optional_arg()
  git-compat-util: introduce skip_to_optional_arg()

Update the shell prompt script (in contrib/) to strip trailing CR
from strings read from various "state" files.

* ra/prompt-eread-fix:
  git-prompt: fix reading files with windows line endings
  git-prompt: make __git_eread intended use explicit

Doc updates.

* bw/path-doc:
  path: document path functions

Hopefully the last one before -rc0
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The build procedure now allows not just the repositories but also
the refs to be used to take pre-formatted manpages and html
documents to install.

* rb/quick-install-doc:
  install-doc-quick: allow specifying what ref to install

Code clean-up.

* jt/transport-no-more-rsync:
  transport: remove unused "push" in vtable

Assorted updates for TravisCI integration.

* sg/travis-fixes:
  travis-ci: use 'set -x' in 'ci/*' scripts for extra tracing output
  travis-ci: set GIT_TEST_HTTPD in 'ci/lib-travisci.sh'
  travis-ci: move setting environment variables to 'ci/lib-travisci.sh'
  travis-ci: introduce a $jobname variable for 'ci/*' scripts

Simplify the ignore rules for t/helper directory.

* sb/test-helper-excludes:
  t/helper: ignore everything but sources

Code clean-up.

* ot/pretty:
  format: create docs for pretty.h
  format: create pretty.h file

Code clean-up.

* bw/submodule-sans-cache-compat:
  submodule: convert get_next_submodule to not rely on the_index
  submodule: used correct index in is_staging_gitmodules_ok
  submodule: convert stage_updated_gitmodules to take a struct index_state

"git clone --shared" to borrow from a (secondary) worktree did not
work, even though "git clone --local" did.  Both are now accepted.

* es/clone-shared-worktree:
  clone: support 'clone --shared' from a worktree

Doc updates.

* tb/delimit-pretty-trailers-args-with-comma:
  docs/pretty-formats: mention commas in %(trailers) syntax

Leakfix.

* rs/fmt-merge-msg-leakfix:
  transport-helper: plug strbuf and string_list leaks

A few structures and variables that are implementation details of
the decorate API have been renamed and then the API got documented
better.

* jt/decorate-api:
  decorate: clean up and document API

Typo/Logico fix.

* jk/cvsimport-quoting:
  cvsimport: apply shell-quoting regex globally