Waaaaaaay back Git was considered to be secure as it never overwrote
an object it already had.  This was ensured by always unpacking the
packfile received over the network (both in fetch and receive-pack)
and our already existing logic to not create a loose object for an
object we already have.

Lately however we keep "large-ish" packfiles on both fetch and push
by running them through index-pack instead of unpack-objects.  This
would let an attacker perform a birthday attack.

How?  Assume the attacker knows a SHA-1 that has two different
data streams.  He knows the client is likely to have the "good"
one.  So he sends the "evil" variant to the other end as part of
a "large-ish" packfile.  The recipient keeps that packfile, and
indexes it.  Now since this is a birthday attack there is a SHA-1
collision; two objects exist in the repository with the same SHA-1.
They have *very* different data streams.  One of them is "evil".

Currently the poor recipient cannot tell the two objects apart,
short of by examining the timestamp of the packfiles.  But lets
say the recipient repacks before he realizes he's been attacked.
We may wind up packing the "evil" version of the object, and deleting
the "good" one.  This is made *even more likely* by Junio's recent
rearrange_packed_git patch (b867092f).

It is extremely unlikely for a SHA1 collisions to occur, but if it
ever happens with a remote (hence untrusted) object we simply must
not let the fetch succeed.

Normally received packs should not contain objects we already have.
But when they do we must ensure duplicated objects with the same SHA1
actually contain the same data.
Signed-off-by: NNicolas Pitre <nico@cam.org>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

This fixes the single force (+) when fetched with fetch_per_ref.

Also use $LF as separator because IFS is $LF.
Signed-off-by: NSanti Béjar <sbejar@gmail.com>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

* git://git2.kernel.org/pub/scm/gitk/gitk:
  [PATCH] gitk: bind <F5> key to Update (reread commits)

We already have -q in git clone.  So for those who care to suppress
the noise during an http based clone, make -q actually do a quiet
http fetch.
Signed-off-by: NChris Wright <chrisw@sous-sol.org>
Cc: Fernando Herrera <fherrera@onirica.com>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

The thing is, if the output buffer is empty, we should *still* actually
use the zlib routines to *unpack* that empty output buffer.

But we had a test that said "only unpack if we still expect more output".

So we wouldn't use up all the zlib stream, because we felt that we didn't
need it, because we already had all the bytes we wanted. And it was
"true": we did have all the output data. We just needed to also eat all
the input data!

We've had this bug before - thinking that we don't need to inflate()
anything because we already had it all..
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

This is a simple but powerful continuous integration build system
for Git.  It works by receiving push events from repositories
through the post-receive hook, aggregates them on a per-branch
basis into a first-come-first-serve build queue, and lets a
background build daemon perform builds one at a time.
Signed-off-by: NShawn O. Pearce <spearce@spearce.org>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

There has not been any work on the shallow stuff lately, so it is hard
to find out what it does, and how. This document describes the ideas
as well as the current problems, and can serve as a starting point for
shallow people.
Signed-off-by: NJohannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

Setting up a git-daemon came up the other day on IRC, and it is slightly
non trivial for the uninitiated.
Signed-off-by: NJohannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

Since in at least one use case, xdl_hash_record() takes over 15% of the
CPU time, it makes sense to even micro-optimize it. For many cases, no
whitespace special handling is needed, and in these cases we should not
even bother to check for whitespace in _every_ iteration of the loop.
Signed-off-by: NJohannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

The commit structures are guaranteed their uniqueness by the object
layer, so we can check their address and see if they are the same
without going down to the object sha1 level.
Signed-off-by: NJunio C Hamano <junkio@cox.net>

Signed-off-by: NJames Bowes <jbowes@dangerouslyinc.com>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

This provides a smoother degradation in performance when the cache
gets trashed due to the delta_base_cache_limit being reached.  Limited
testing with really small delta_base_cache_limit values appears to confirm
this.
Signed-off-by: NNicolas Pitre <nico@cam.org>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

Currently there are 3 different ways to deal with the cache size.
Let's stick to only one.  The compiler is smart enough to produce the exact
same code in those cases anyway.
Signed-off-by: NNicolas Pitre <nico@cam.org>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

I think we can start to slow down, as we now have covered
everything I listed earlier in the short-term release plan.

The last release 1.5.0 took painfully too long.
Signed-off-by: NJunio C Hamano <junkio@cox.net>

An earlier conversion to run_command() from execlp() forgot that
run_command() takes an array that is terminated with NULL.
Signed-off-by: NJunio C Hamano <junkio@cox.net>

This is mainly just a cleanup patch, and sets up for later changes where
the tree-diff.c "interesting()" function can return more than just a
yes/no value.

In particular, it should be quite possible to say "no subsequent entries
in this tree can possibly be interesting any more", and thus allow the
callers to short-circuit the tree entirely.

In fact, changing the callers to do so is trivial, and is really all this
patch really does, because changing "interesting()" itself to say that
nothing further is going to be interesting is definitely more complicated,
considering that we may have arbitrary pathspecs.

But in cleaning up the callers, this actually fixes a potential small
performance issue in diff_tree(): if the second tree has a lot of
uninterestign crud in it, we would keep on doing the "is it interesting?"
check on the first tree for each uninteresting entry in the second one.

The answer is obviously not going to change, so that was just not helping.
The new code is clearer and simpler and avoids this issue entirely.

I also renamed "interesting()" to "tree_entry_interesting()", because I
got frustrated by the fact that

 - we actually had *another* function called "interesting()" in another
   file, and I couldn't tell from the profiles which one was the one that
   mattered more.

 - when rewriting it to return a ternary value, you can't just do

	if (interesting(...))
		...

   any more, but want to assign the return value to a local variable. The
   name of choice for that variable would normally be "interesting", so
   I just wanted to make the function name be more specific, and avoid
   that whole issue (even though I then didn't choose that name for either
   of the users, just to avoid confusion in the patch itself ;)

In other words, this doesn't really change anything, but I think it's a
good thing to do, and if somebody comes along and writes the logic for
"yeah, none of the pathspecs you have are interesting", we now support
that trivially.

It could easily be a meaningful optimization for things like "blame",
where there's just one pathspec, and stopping when you've seen it would
allow you to avoid about 50% of the tree traversals on average.
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

This makes "track_tree_refs()" use the same "tree_entry()" function for
counting the entries as it does for actually traversing them a few lines
later.

Not a biggie, but the reason I care was that this was the only user of
"update_tree_entry()" that didn't actually *extract* the tree entry first.
It doesn't matter as things stand now, but it meant that a separate
test-patch I had that avoided a few more "strlen()" calls by just saving
the entry length in the entry descriptor and using it directly when
updating wouldn't work without this patch.
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

Run the pre-commit and post-commit hooks at appropriate places, and
display their output if any.
Signed-off-by: NAlexandre Julliard <julliard@winehq.org>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

* jb/gc:
  Make gc a builtin.

* fl/cvsserver:
  cvsserver: further improve messages on commit and status
  cvsserver: Be more chatty

The new configuration variable core.deltaBaseCacheLimit allows the
user to control how much memory they are willing to give to Git for
caching base objects of deltas.  This is not normally meant to be
a user tweakable knob; the "out of the box" settings are meant to
be suitable for almost all workloads.

We default to 16 MiB under the assumption that the cache is not
meant to consume all of the user's available memory, and that the
cache's main purpose was to cache trees, for faster path limiters
during revision traversal.  Since trees tend to be relatively small
objects, this relatively small limit should still allow a large
number of objects.

On the other hand we don't want the cache to start storing 200
different versions of a 200 MiB blob, as this could easily blow
the entire address space of a 32 bit process.

We evict OBJ_BLOB from the cache first (credit goes to Junio) as
we want to favor OBJ_TREE within the cache.  These are the objects
that have the highest inflate() startup penalty, as they tend to
be small and thus don't have that much of a chance to ammortize
that penalty over the entire data.
Signed-off-by: NShawn O. Pearce <spearce@spearce.org>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

* sp/run-command:
  Use run_command within send-pack
  Use run_command within receive-pack to invoke index-pack
  Use run_command within merge-index
  Use run_command for proxy connections
  Use RUN_GIT_CMD to run push backends
  Correct new compiler warnings in builtin-revert
  Replace fork_with_pipe in bundle with run_command
  Teach run-command to redirect stdout to /dev/null
  Teach run-command about stdout redirection

In the Linux kernel, for example, it's common to include Cc: lines
for cases when you want to remember to cc someone on a patch without
necessarily claiming they signed off on it.  Make git-send-email
aware of these.
Signed-off-by: N"J. Bruce Fields" <bfields@citi.umich.edu>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

Also add support for vimdiff
Signed-off-by: N"Theodore Ts'o" <tytso@mit.edu>

Signed-off-by: NJames Bowes <jbowes@dangerouslyinc.com>
Signed-off-by: N"Theodore Ts'o" <tytso@mit.edu>

Signed-off-by: NJunio C Hamano <junkio@cox.net>

* ar/diff:
  Add tests for --quiet option of diff programs
  try-to-simplify-commit: use diff-tree --quiet machinery.
  revision.c: explain what tree_difference does
  Teach --quiet to diff backends.
  diff --quiet
  Remove unused diffcore_std_no_resolve
  Allow git-diff exit with codes similar to diff(1)

This is a micro-optimization that grew out of the mailing list discussion
about "strlen()" showing up in profiles.

We used to pass regular C strings around to the low-level tree walking
routines, and while this worked fine, it meant that we needed to call
strlen() on strings that the caller always actually knew the size of
anyway.

So pass the length of the string down wih the string, and avoid
unnecessary calls to strlen(). Also, when extracting a pathname from a
tree entry, use "tree_entry_len()" instead of strlen(), since the length
of the pathname is directly calculable from the decoded tree entry itself
without having to actually do another strlen().

This shaves off another ~5-10% from some loads that are very tree
intensive (notably doing commit filtering by a pathspec).

Signed-off-by: Linus Torvalds  <torvalds@linux-foundation.org>"
Signed-off-by: NJunio C Hamano <junkio@cox.net>

A malloc() + memcpy() will always be faster than mmap() +
malloc() + inflate().  If the data is already there it is
certainly better to copy it straight away.

With this patch below I can do 'git log drivers/scsi/ >
/dev/null' about 7% faster.  I bet it might be even more on
those platforms with bad mmap() support.
Signed-off-by: NNicolas Pitre <nico@cam.org>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

This trivial 256-entry delta_base cache improves performance for some
loads by a factor of 2.5 or so.

Instead of always re-generating the delta bases (possibly over and over
and over again), just cache the last few ones. They often can get re-used.
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

This doesn't change any code, it just creates a point for where we'd
actually do the caching of delta bases that have been generated.
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

Signed-off-by: NJunio C Hamano <junkio@cox.net>

Signed-off-by: NJames Bowes <jbowes@dangerouslyinc.com>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

* maint:
  git-merge: finish when git-read-tree fails

Especially with the new index format to come, it is more appropriate
to encapsulate more into check_packed_git_idx() and assume less of the
index format in struct packed_git.

To that effect, the index_base is renamed to index_data with void * type
so it is not used directly but other pointers initialized with it. This
allows for a couple pointer cast removal, as well as providing a better
generic name to grep for when adding support for new index versions or
formats.

And index_data is declared const too while at it.
Signed-off-by: NNicolas Pitre <nico@cam.org>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

Make sure pack-objects with --delta-base-offset works fine, and that
it actually produces smaller packs as expected.
Signed-off-by: NNicolas Pitre <nico@cam.org>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

The 'use packed deltified objects' test was flawed as it failed to
remove the pack and index from the previous test, effectively preventing
the desired pack from being exercised as objects could be found in that
other pack instead.
Signed-off-by: NNicolas Pitre <nico@cam.org>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

Signed-off-by: NNicolas Pitre <nico@cam.org>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

An earlier patch 87ab7992 broke applymbox by blindly copying piece
from git-am, causing a harmless but annoying series of error messages.
Signed-off-by: NJunio C Hamano <junkio@cox.net>