When transmitting and receiving POSTs for protocol v0 and v1,
remote-curl uses post_rpc() (and associated functions), but when doing
the same for protocol v2, it uses a separate set of functions
(proxy_rpc() and others). Besides duplication of code, this has caused
at least one bug: the auth retry mechanism that was implemented in v0/v1
was not implemented in v2.

To fix this issue and avoid it in the future, make remote-curl also use
post_rpc() when handling protocol v2. Because line lengths are written
to the HTTP request in protocol v2 (unlike in protocol v0/v1), this
necessitates changes in post_rpc() and some of the functions it uses;
perform these changes too.

A test has been included to ensure that the code for both the unchunked
and chunked variants of the HTTP request is exercised.

Note: stateless_connect() has been updated to use the lower-level packet
reading functions instead of struct packet_reader. The low-level control
is necessary here because we cannot change the destination buffer of
struct packet_reader while it is being used; struct packet_buffer has a
peeking mechanism which relies on the destination buffer being present
in between a peek and a read.
Signed-off-by: NJonathan Tan <jonathantanmy@google.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Currently, whenever remote-curl reads pkt-lines from its response file
descriptor, only the payload is written to its buf, not the 4 characters
denoting the length. A future patch will require the ability to also
write those 4 characters, so in preparation for that, refactor this read
into its own function.
Signed-off-by: NJonathan Tan <jonathantanmy@google.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The result field in struct rpc_state is only used in rpc_service(), and
not in any functions it directly or indirectly calls. Refactor it to
become an argument of rpc_service() instead.
Signed-off-by: NJonathan Tan <jonathantanmy@google.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The stdin_preamble field in struct rpc_state is only used in
rpc_service(), and not in any functions it directly or indirectly calls.
Refactor it to become an argument of rpc_service() instead.

An observation of all callers of rpc_service() shows that the preamble
is always set, so we no longer need the "if (preamble)" check.
Signed-off-by: NJonathan Tan <jonathantanmy@google.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The argv field in struct rpc_state is only used in rpc_service(), and
not in any functions it directly or indirectly calls. Refactor it to
become an argument of rpc_service() instead.
Signed-off-by: NJonathan Tan <jonathantanmy@google.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Signed-off-by: NJunio C Hamano <gitster@pobox.com>

* ab/rebase-test-fix:
  rebase: fix regression in rebase.useBuiltin=false test mode

* rb/no-dev-zero-in-test:
  t5562: replace /dev/zero with a pipe from generate_zero_bytes
  t5318: replace use of /dev/zero with generate_zero_bytes
  test-lib-functions.sh: add generate_zero_bytes function

* rs/bash-is-in-coreutils-on-nonstop:
  config.mak.uname: move location of bash on NonStop to CoreUtils

Windows update.

* js/mingw-host-cpu:
  mingw: use a more canonical method to fix the CPU reporting

Test improvement.

* sg/stress-test:
  test-lib: fix non-portable pattern bracket expressions
  test-lib: make '--stress' more bisect-friendly

Test fix.

* kd/t0028-octal-del-is-377-not-777:
  t0028: fix wrong octal values for BOM in setup

The code and tests assume that the system supplied iconv() would
always use BOM in its output when asked to encode to UTF-16 (or
UTF-32), but apparently some implementations output big-endian
without BOM.  A compile-time knob has been added to help such
systems (e.g. NonStop) to add BOM to the output to increase
portability.

* bc/utf16-portability-fix:
  utf8: handle systems that don't write BOM for UTF-16

* nd/fileno-may-be-macro:
  git-compat-util: work around fileno(fp) that is a macro

* nd/get-oid-with-context-returns-an-enum:
  get_oid_with_context(): match prototype and implementation

Code clean-up.

* rj/sequencer-sign-off-header-static:
  sequencer: make sign_off_header a file local symbol

Fix a recently introduced regression in c762aada ("rebase -x: sanity
check command", 2019-01-29) triggered when running the tests with
GIT_TEST_REBASE_USE_BUILTIN=false. See 62c23938 ("tests: add a
special setup where rebase.useBuiltin is off", 2018-11-14) for how
that test mode works.

As discussed on-list[1] it's not worth it to implement the sanity
check in the legacy rebase code, we plan to remove it after the 2.21
release. So let's do the bare minimum to make the tests pass under the
GIT_TEST_REBASE_USE_BUILTIN=false special setup.

1. https://public-inbox.org/git/xmqqva1nbeno.fsf@gitster-ct.c.googlers.com/Signed-off-by: NÆvar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

In `git version --build-options`, we report also the CPU, but in Git for
Windows we actually cross-compile the 32-bit version in a 64-bit Git for
Windows, so we cannot rely on the auto-detected value.

In 3815f64b (mingw: fix CPU reporting in `git version
--build-options`, 2019-02-07), we fixed this by a Windows-only
workaround, making use of magic pre-processor constants, which works in
GCC, but most likely not all C compilers.

As pointed out by Eric Sunshine, there is a better way, anyway: to set
the Makefile variable HOST_CPU explicitly for cross-compiled Git. So
let's do that!

This reverts commit 3815f64b partially.
Signed-off-by: NJohannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The default bash is now officially in /usr/coreutils/bin instead
of in /usr/local/bin. This version of bash is more stable and
recommended for all use as of the J06.22 and L18.02 operating
system revision levels. This new version provides more stability
of test results.
Signed-off-by: NRandall S. Becker <rsbecker@nexbridge.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

To help platforms that lack /dev/zero (e.g. NonStop), replace use
of /dev/zero to feed "git http-backend" with a pipe of output from
the generate_zero_bytes helper.
Signed-off-by: NRandall S. Becker <rsbecker@nexbridge.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

There are platforms (e.g. NonStop) that lack /dev/zero; use the
generate_zero_bytes helper we just introduced to append stream
of NULs at the end of the file.

The original, even though it uses "dd seek=... count=..." to make it
look like it is overwriting the middle part of an existing file, has
truncated the file before this step with another use of "dd", which
may make it tricky to see why this rewrite is a correct one.
Signed-off-by: NRandall S. Becker <rsbecker@nexbridge.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The get_oid_with_context() function is declared to return an enum in
cache.h, but defined to return an int in sha1-name.c.  The compiler
notices this on AIX and rejects the build, since d1dd94b3 (Do not
print 'dangling' for cat-file in case of ambiguity - 2019-01-17) was
merged.

Return the correct type from the implementation to fix this.
Signed-off-by: NNguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

On various BSD's, fileno(fp) is implemented as a macro that directly
accesses the fields in the FILE * object, which breaks a function that
accepts a "void *fp" parameter and calls fileno(fp) and expect it to
work.

Work it around by adding a compile-time knob FILENO_IS_A_MACRO that
inserts a real helper function in the middle of the callchain.
Signed-off-by: NNguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

t5318 and t5562 used /dev/zero, which is not portable. This function
provides both a fixed block of NUL bytes and an infinite stream of NULs.
Signed-off-by: NRandall S. Becker <rsbecker@nexbridge.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

These are just some small fixes I noticed doing a complete read-through
(there are a few cases I left that are incomplete or abbreviated
sentences, but I think those are OK in this sort of bullet-list style).
Signed-off-by: NJeff King <peff@peff.net>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

In the feature that was eventually committed, "--date=auto" doesn't do
anything. It was generalized to "--date=auto:<format>".
Signed-off-by: NJeff King <peff@peff.net>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

* nd/imap-send-typofix:
  imap-send.c: add a missing space in error message

Signed-off-by: NNguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

When serializing UTF-16 (and UTF-32), there are three possible ways to
write the stream. One can write the data with a BOM in either big-endian
or little-endian format, or one can write the data without a BOM in
big-endian format.

Most systems' iconv implementations choose to write it with a BOM in
some endianness, since this is the most foolproof, and it is resistant
to misinterpretation on Windows, where UTF-16 and the little-endian
serialization are very common. For compatibility with Windows and to
avoid accidental misuse there, Git always wants to write UTF-16 with a
BOM, and will refuse to read UTF-16 without it.

However, musl's iconv implementation writes UTF-16 without a BOM,
relying on the user to interpret it as big-endian. This causes t0028 and
the related functionality to fail, since Git won't read the file without
a BOM.

Add a Makefile and #define knob, ICONV_OMITS_BOM, that can be set if the
iconv implementation has this behavior. When set, Git will write a BOM
manually for UTF-16 and UTF-32 and then force the data to be written in
UTF-16BE or UTF-32BE. We choose big-endian behavior here because the
tests use the raw "UTF-16" encoding, which will be big-endian when the
implementation requires this knob to be set.

Update the tests to detect this case and write test data with an added
BOM if necessary. Always write the BOM in the tests in big-endian
format, since all iconv implementations that omit a BOM must use
big-endian serialization according to the Unicode standard.

Preserve the existing behavior for systems which do not have this knob
enabled, since they may use optimized implementations, including
defaulting to the native endianness, which may improve performance.
Signed-off-by: Nbrian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The setup code uses octal values with printf to generate a BOM for
UTF-16/32 BE/LE. It specifically uses '\777' to emit a 0xff byte. This
relies on the fact that most shells truncate the value above 0o377.

Ash however interprets '\777' as '\77' + a literal '7', resulting in an
invalid BOM.

Fix this by using the proper value of 0xff: '\377'.
Signed-off-by: NKevin Daudt <me@ikke.info>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Use a '!' character to start a non-matching pattern bracket
expression, as specified by POSIX in Shell Command Language section
2.13.1 Patterns Matching a Single Character [1].

I used '^' instead in three places in the previous three commits, to
verify that the arguments of the '--stress=' and '--stress-limit='
options and the values of various '*_PORT' environment variables are
valid numbers.  With certain shells, at least with dash (upstream and
in Ubuntu 14.04) and mksh, this led to various undesired behaviors:

  # error message in case of a valid number
  $ ~/src/dash/src/dash ./t3903-stash.sh --stress=8
  error: --stress=<N> requires the number of jobs to run

  # not the expected error message
  $ ~/src/dash/src/dash ./t3903-stash.sh --stress=foo
  ./t3903-stash.sh: 238: test: Illegal number: foo

  # no error message at all?!
  $ mksh ./t3903-stash.sh --stress=foo
  $ echo $?
  0

Some other shells, e.g. Bash (even in posix mode), ksh, dash in Ubuntu
16.04 or later, are apparently happy to accept '^' just as well.

[1] http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_13Signed-off-by: NSZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Commit d0aaa46f ("commit: move empty message checks to libgit",
2017-11-10) removes the last use of 'sign_off_header' outside of
the "sequencer.c" source file. Remove the extern declaration from
the header file and mark the definition of the symbol with the
static keyword.
Signed-off-by: NRamsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The NonStop platform needs this configuration item specified as
UnfortunatelyYes so that config directory files are correctly processed.
Signed-off-by: NRandall S. Becker <rsbecker@nexbridge.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Signed-off-by: NJean-Noël Avila <jn.avila@free.fr>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Windows update.

* js/mingw-host-cpu:
  mingw: fix CPU reporting in `git version --build-options`

Update to the fuzzer.

* js/fuzz-commit-graph-update:
  object: fix leak of shallow_stat
  fuzz-commit-graph: initialize repo object

Doc update.

* kl/pretty-doc-markup-fix:
  doc: prevent overflowing <code> tag in rendered HTML

Build update.

* sg/ci-parallel-build:
  ci: clear and mark MAKEFLAGS exported just once
  ci: make sure we build Git parallel

A flakey "p4" test has been removed.

* ld/git-p4-remove-flakey-test:
  git-p4: remove ticket expiry test