The packets sent during ref negotiation are all terminated
by newline; even though the code to chomp these newlines is
short, we end up doing it in a lot of places.

This patch teaches packet_read_line to auto-chomp the
trailing newline; this lets us get rid of a lot of inline
chomping code.

As a result, some call-sites which are not reading
line-oriented data (e.g., when reading chunks of packfiles
alongside sideband) transition away from packet_read_line to
the generic packet_read interface. This patch converts all
of the existing callsites.

Since the function signature of packet_read_line does not
change (but its behavior does), there is a possibility of
new callsites being introduced in later commits, silently
introducing an incompatibility.  However, since a later
patch in this series will change the signature, such a
commit would have to be merged directly into this commit,
not to the tip of the series; we can therefore ignore the
issue.

This is an internal cleanup and should produce no change of
behavior in the normal case. However, there is one corner
case to note. Callers of packet_read_line have never been
able to tell the difference between a flush packet ("0000")
and an empty packet ("0004"), as both cause packet_read_line
to return a length of 0. Readers treat them identically,
even though Documentation/technical/protocol-common.txt says
we must not; it also says that implementations should not
send an empty pkt-line.

By stripping out the newline before the result gets to the
caller, we will now treat the newline-only packet ("0005\n")
the same as an empty packet, which in turn gets treated like
a flush packet. In practice this doesn't matter, as neither
empty nor newline-only packets are part of git's protocols
(at least not for the line-oriented bits, and readers who
are not expecting line-oriented packets will be calling
packet_read directly, anyway). But even if we do decide to
care about the distinction later, it is orthogonal to this
patch.  The right place to tighten would be to stop treating
empty packets as flush packets, and this change does not
make doing so any harder.
Signed-off-by: NJeff King <peff@peff.net>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The current parsing scheme for upload-archive is to pack
arguments into a fixed-size buffer, separated by NULs, and
put a pointer to each argument in the buffer into a
fixed-size argv array.

This works fine, and the limits are high enough that nobody
reasonable is going to hit them, but it makes the code hard
to follow.  Instead, let's just stuff the arguments into an
argv_array, which is much simpler. That lifts the "all
arguments must fit inside 4K together" limit.

We could also trivially lift the MAX_ARGS limitation (in
fact, we have to keep extra code to enforce it). But that
would mean a client could force us to allocate an arbitrary
amount of memory simply by sending us "argument" lines. By
limiting the MAX_ARGS, we limit an attacker to about 4
megabytes (64 times a maximum 64K packet buffer). That may
sound like a lot compared to the 4K limit, but it's not a
big deal compared to what git-archive will actually allocate
while working (e.g., to load blobs into memory). The
important thing is that it is bounded.
Signed-off-by: NJeff King <peff@peff.net>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

According to the comment, enter_repo will modify its input.
However, this has not been the case since 1c64b48e
(enter_repo: do not modify input, 2011-10-04). Drop the
now-useless copy.
Signed-off-by: NJeff King <peff@peff.net>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The POSIX-function fork is not supported on Windows. Use our
start_command API instead, respawning ourselves in a special
"writer" mode to follow the alternate code path.

Remove the NOT_MINGW-prereq for t5000, as git-archive --remote
now works.
Signed-off-by: NJeff King <peff@peff.net>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

This reverts commit c09cd77e, expecting a
better version to be rerolled soon.

The POSIX-function fork is not supported on Windows. Use our
start_command API instead.

As this is the last call-site that depends on the fork-stub in
compat/mingw.h, remove that as well.

Add an undocumented flag to git-archive that tells it that the
action originated from a remote, so features can be disabled.
Thanks to Jeff King for work on this part.

Remove the NOT_MINGW-prereq for t5000, as git-archive --remote
now works.
Signed-off-by: NErik Faye-Lund <kusmabite@gmail.com>
Helped-by: NJeff King <peff@peff.net>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Some tar filters may be very expensive to run, so sites do
not want to expose them via upload-archive. This patch lets
users configure tar.<filter>.remote to turn them off.

By default, gzip filters are left on, as they are about as
expensive as creating zip archives.
Signed-off-by: NJeff King <peff@peff.net>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The process for guessing an archive output format based on
the filename is something like this:

  a. parse --output in cmd_archive; check the filename
     against a static set of mapping heuristics (right now
     it just matches ".zip" for zip files).

  b. if found, stick a fake "--format=zip" at the beginning
     of the arguments list (if the user did specify a
     --format manually, the later option will override our
     fake one)

  c. if it's a remote call, ship the arguments to the remote
     (including the fake), which will call write_archive on
     their end

  d. if it's local, ship the arguments to write_archive
     locally

There are two problems:

  1. The set of mappings is static and at too high a level.
     The write_archive level is going to check config for
     user-defined formats, some of which will specify
     extensions. We need to delay lookup until those are
     parsed, so we can match against them.

  2. For a remote archive call, our set of mappings (or
     formats) may not match the remote side's. This is OK in
     practice right now, because all versions of git
     understand "zip" and "tar". But as new formats are
     added, there is going to be a mismatch between what the
     client can do and what the remote server can do.

To fix (1), this patch refactors the location guessing to
happen at the write_archive level, instead of the
cmd_archive level. So instead of sticking a fake --format
field in the argv list, we actually pass a "name hint" down
the callchain; this hint is used at the appropriate time to
guess the format (if one hasn't been given already).

This patch leaves (2) unfixed. The name_hint is converted to
a "--format" option as before, and passed to the remote.
This means the local side's idea of how extensions map to
formats will take precedence.

Another option would be to pass the name hint to the remote
side and let the remote choose. This isn't a good idea for
two reasons:

  1. There's no room in the protocol for passing that
     information. We can pass a new argument, but older
     versions of git on the server will choke on it.

  2. Letting the remote side decide creates a silent
     inconsistency in user experience. Consider the case
     that the locally installed git knows about the "tar.gz"
     format, but a remote server doesn't.

     Running "git archive -o foo.tar.gz" will use the tar.gz
     format. If we use --remote, and the local side chooses
     the format, then we send "--format=tar.gz" to the
     remote, which will complain about the unknown format.
     But if we let the remote side choose the format, then
     it will realize that it doesn't know about "tar.gz" and
     output uncompressed tar without even issuing a warning.
Signed-off-by: NJeff King <peff@peff.net>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

This shrinks the top-level directory a bit, and makes it much more
pleasant to use auto-completion on the thing. Instead of

	[torvalds@nehalem git]$ em buil<tab>
	Display all 180 possibilities? (y or n)
	[torvalds@nehalem git]$ em builtin-sh
	builtin-shortlog.c     builtin-show-branch.c  builtin-show-ref.c
	builtin-shortlog.o     builtin-show-branch.o  builtin-show-ref.o
	[torvalds@nehalem git]$ em builtin-shor<tab>
	builtin-shortlog.c  builtin-shortlog.o
	[torvalds@nehalem git]$ em builtin-shortlog.c

you get

	[torvalds@nehalem git]$ em buil<tab>		[type]
	builtin/   builtin.h
	[torvalds@nehalem git]$ em builtin		[auto-completes to]
	[torvalds@nehalem git]$ em builtin/sh<tab>	[type]
	shortlog.c     shortlog.o     show-branch.c  show-branch.o  show-ref.c     show-ref.o
	[torvalds@nehalem git]$ em builtin/sho		[auto-completes to]
	[torvalds@nehalem git]$ em builtin/shor<tab>	[type]
	shortlog.c  shortlog.o
	[torvalds@nehalem git]$ em builtin/shortlog.c

which doesn't seem all that different, but not having that annoying
break in "Display all 180 possibilities?" is quite a relief.

NOTE! If you do this in a clean tree (no object files etc), or using an
editor that has auto-completion rules that ignores '*.o' files, you
won't see that annoying 'Display all 180 possibilities?' message - it
will just show the choices instead.  I think bash has some cut-off
around 100 choices or something.

So the reason I see this is that I'm using an odd editory, and thus
don't have the rules to cut down on auto-completion.  But you can
simulate that by using 'ls' instead, or something similar.
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

We already have these checks in many printf-type functions that have
prototypes which are in header files.  Add these same checks to some
more prototypes in header functions and to static functions in .c
files.

cc: Miklos Vajna <vmiklos@frugalware.org>
Signed-off-by: NTarmigan Casebolt <tarmigan+git@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

In theory it is possible for sideband channel #2 to be delayed if
pack data is quick to come up for sideband channel #1.  And because
data for channel #2 is read only 128 bytes at a time while pack data
is read 8192 bytes at a time, it is possible for many pack blocks to
be sent to the client before the progress message fifo is emptied,
making the situation even worse.  This would result in totally garbled
progress display on the client's console as local progress gets mixed
with partial remote progress lines.

Let's prevent such situations by giving transmission priority to
progress messages over pack data at all times.
Signed-off-by: NNicolas Pitre <nico@fluxnic.net>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

On Cygwin, poll() reports POLLIN even for file descriptors that have
reached their end.  This caused git upload-archive to be stuck in an
infinite loop, as it only looked at the POLLIN flag.

In addition to POLLIN, check if read() returned 0, which indicates
end-of-file, and keep looping only as long as at least one of the file
descriptors has input.  This lets the following command finish on its
own when run in a git repository on Cygwin, instead of it getting stuck
after printing all file names:

	$ git archive -v --remote . HEAD >/dev/null
Reported-by: NBob Kagy <bobkagy@gmail.com>
Signed-off-by: NRene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Certain remote commands, when asked to do something in a
particular directory that was not actually a git repository,
would say "unable to chdir or not a git archive". The
"chdir" bit is an unnecessary detail, and the term "git
archive" is much less common these days than "git repository".

So let's switch them all to:

  fatal: '%s' does not appear to be a git repository
Signed-off-by: NJeff King <peff@peff.net>
Acked-by: NShawn O. Pearce <spearce@spearce.org>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

MAX_EXTRA_ARGS is not used anymore, so remove it.  MAX_ARGS is used only
in builtin-upload-archive.c, so define it there.  Also report the actual
value we're comparing against when the number of args is too big.
Signed-off-by: NRene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Both archive and upload-archive have to parse command line arguments and
then call the archiver specific write function.  Move the duplicate code
to a new function, write_archive().
Signed-off-by: NRene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Pass struct archiver and struct archiver_args explicitly to parse_archive_args
and remove the latter from the former.  This allows us to get rid of struct
archiver_desc and simplifies the code a bit.
Signed-off-by: NRene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

When you misuse a git command, you are shown the usage string.
But this is currently shown in the dashed form.  So if you just
copy what you see, it will not work, when the dashed form
is no longer supported.

This patch makes git commands show the dash-less version.

For shell scripts that do not specify OPTIONS_SPEC, git-sh-setup.sh
generates a dash-less usage string now.
Signed-off-by: NStephan Beyer <s-beyer@gmx.net>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Make sure that buf has enough space to store the trailing \0 of
the command line argument, too.
Signed-off-by: NJochen Voss <voss@seehuhn.de>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

We have a number of badly checked read() calls.  Often we are
expecting read() to read exactly the size we requested or fail, this
fails to handle interrupts or short reads.  Add a read_in_full()
providing those semantics.  Otherwise we at a minimum need to check
for EINTR and EAGAIN, where this is appropriate use xread().
Signed-off-by: NAndy Whitcroft <apw@shadowen.org>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

This is a mechanical clean-up of the way *.c files include
system header files.

 (1) sources under compat/, platform sha-1 implementations, and
     xdelta code are exempt from the following rules;

 (2) the first #include must be "git-compat-util.h" or one of
     our own header file that includes it first (e.g. config.h,
     builtin.h, pkt-line.h);

 (3) system headers that are included in "git-compat-util.h"
     need not be included in individual C source files.

 (4) "git-compat-util.h" does not have to include subsystem
     specific header files (e.g. expat.h).
Signed-off-by: NJunio C Hamano <junkio@cox.net>

Looks like ctype again. Gotta be careful with that on BSD releases:

    $ gmake prefix=/opt/git all
    GIT_VERSION = 1.4.2.GIT
    gcc -o builtin-upload-archive.o -c -g -O2 -Wall -I/usr/local/include -DSHA1_HEADER='<openssl/sha.h>' -DNO_STRCASESTR builtin-upload-archive.c
    In file included from /usr/include/sys/poll.h:54,
                     from builtin-upload-archive.c:11:
    /usr/include/ctype.h:68: error: syntax error before ']' token
    /usr/include/ctype.h:69: error: syntax error before ']' token
    ...
    /usr/include/sys/poll.h:53:1: unterminated #ifndef
    /usr/include/sys/poll.h:28:1: unterminated #ifndef
    gmake: *** [builtin-upload-archive.o] Error 1

This fixes it.
Signed-off-by: NJunio C Hamano <junkio@cox.net>

The current code works like this: if others flags than POLLIN is
raised we assume that (a) something bad happened and the child died or
(b) the child has closed the pipe because it had no more data to send.

For the latter case, we assume wrongly that one call to
process_input() will empty the pipe. Indeed it reads only 16Ko of data
by call and the the pipe capacity can be larger than that (on current
Linux kernel, it is 65536 bytes). Therefore the child can write 32ko
of data, for example, and close the pipe. After that poll will return
POLLIN _and_ POLLHUP and the parent will read only 16ko of data.

This patch forces the parent to empty the pipe as soon as POLLIN is
raised and even if POLLHUP or something else is raised too.

Moreover, some implementations of poll might return POLLRDNORM flag
even if it is non standard.
Signed-off-by: NFranck Bui-Huu <vagabon.xyz@gmail.com>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

Franck noticed that the code around polling and relaying messages
from the child process was quite bogus.  Here is an attempt to
clean it up a bit, based on his patch:

 - When POLLHUP is set, it goes ahead and reads the file
   descriptor.  Worse yet, it does not check the return value of
   read() for errors when it does.

 - When we processed one POLLIN, we should just go back and see
   if any more data is available.  We can check if the child is
   still there when poll gave control back at us but without any
   actual input.

[jc: with simplification suggested by Franck. ]
Signed-off-by: NJunio C Hamano <junkio@cox.net>

Using the refactored sideband code from existing upload-pack protocol,
this lets the error condition and status output sent from the remote
process to be shown locally.
Signed-off-by: NJunio C Hamano <junkio@cox.net>

This command implements the git archive protocol on the server
side. This command is not intended to be used by the end user.
Underlying git-archive command line options are sent over the
protocol from "git-archive --remote=...", just like upload-tar
currently does with "git-tar-tree=...".

As for "git-archive" command implementation, this new command
does not execute any existing "git-{tar,zip}-tree" but rely
on the archive API defined by "git-archive" patch. Hence we
get 2 good points:

 - "git-archive" and "git-upload-archive" share all option
   parsing code.

 - All kind of git-upload-{tar,zip} can be deprecated.
Signed-off-by: NFranck Bui-Huu <vagabon.xyz@gmail.com>
Signed-off-by: NJunio C Hamano <junkio@cox.net>