Like git-receive-pack, this is only partway done.

It's not working yet, but it's at the point where I want to be able to
track my changes.  The theory of operation is that this is the "remote"
side of a "git push".  It can tell us what references the remote side
has, receives out reference update commands and a pack-file, and can
execute the unpacking command.

It makes gitk unhappy, and besides, non-merge-order doesn't complain, so
why do it here..

Now you can give git-rev-list tags, trees and blobs, and it will do the
proper reachability for them all. Knock wood.

Of course, you need the "--objects" flag to do anything but plain
commits.

We want to be able to just say "give a difference between these
objects", rather than limiting it to commits only.  This isn't there
yet, but it sets things up to be a bit easier.

It now always read from standard input and rejects non-flag
arguments.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

It can no longer be as verbose, since it doesn't have a good way to
resolve deltas (now that it is purely streaming, it cannot seek around
to read the objects a delta is based on).

But it can check that the thing unpacks cleanly at least as far as pack
syntax goes - all the objects uncompress cleanly, and the pack has the
right final SHA1.

I'd like to add back the "dry-run" thing, but it turns out that to do it
well, I'd have to keep all the object data in memory (which is not
acceptable).  So I'll clean it up a bit and make it do as many checks as
it can.

Given a list of <pack>.idx files, this command validates the
index file and the corresponding .pack file for consistency.

This patch also uses the same validation mechanism in fsck-cache
when the --full flag is used.

During normal operation, sha1_file.c verifies that a given .idx
file matches the .pack file by comparing the SHA1 checksum
stored in .idx file and .pack file as a minimum sanity check.
We may further want to check the pack signature and version when
we map the pack, but that would be a separate patch.

Earlier, errors to map a pack file was not flagged fatal but led
to a random fatal error later.  This version explicitly die()s
when such an error is detected.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

If we prefer 0 as maxsize for diff_delta() to say "unlimited", let's be
consistent about it.

This patch also fixes type mismatch in a call to get_delta_hdr_size()
from packed_delta_info().
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

This is a wrap-up patch including all the cleanups I've done to the
delta code and its usage.  The most important change is the
factorization of the delta header handling code.
Signed-off-by: NNicolas Pitre <nico@cam.org>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Pointed out by Junio.

This makes it match the new delta encoding, and admittedly makes the
code easier to follow.

This also updates the PACK file version to 2, since this (and the delta
encoding change in the previous commit) are incompatible with the old
format.

Since the delta data format is not tied to any actual git object
anymore, now is the time to add a small improvement to the delta data
header as it is been done for packed object header.  This patch allows
for reducing the delta header of about 2 bytes and makes for simpler
code.
Signed-off-by: NNicolas Pitre <nico@cam.org>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Deltas are useless by themselves and when you use them you need to get
to their base objects.  A base object should inherit recency from the
most recent deltified object that is based on it and that is what this
patch teaches git-pack-objects.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

It gets a bit more complicated to unpack in a streaming environment, but
here it is.  The rewrite is actually a lot cleaner in other ways, it's
just a bit more subtle.

The diff_delta() interface was extended to reject generating too big a
delta while we were working on the packed GIT archive format.

Take advantage of that when generating delta in the similarity estimator
used in diffcore-rename.c
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Standalone unpack-objects command was not adjusted for header length
encoding change when dealing with deltified entry.  This fixes it.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

It's just easier to let git help out with the merge than it is to try to
fix up the diffs.

The fsck-cache complains if objects referred to by files in .git/refs/
or objects stored in files under .git/objects/??/ are not found as
stand-alone SHA1 files (i.e.  found in alternate object pools
GIT_ALTERNATE_OBJECT_DIRECTORIES or packed archives stored under
.git/objects/pack).

Although this is a good semantics to maintain consistency of a single
.git/objects directory as a self contained set of objects, it sometimes
is useful to consider it is OK as long as these "outside" objects are
available.

This commit introduces a new flag, --standalone, to git-fsck-cache.
When it is not specified, connectivity checks and .git/refs pointer
checks are taught that it is OK when expected objects do not exist under
.git/objects/?? hierarchy but are available from an packed archive or in
an alternate object pool.

Another new flag, --full, makes git-fsck-cache to check not only the
current GIT_OBJECT_DIRECTORY but also objects found in alternate object
pools and packed GIT archives.a
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

The commands git-fsck-cache and probably git-*-pull needs to have a way
to enumerate objects contained in packed GIT archives and alternate
object pools.  This commit exposes the data structure used to keep track
of them from sha1_file.c, and adds a couple of accessor interface
functions for use by the enhanced git-fsck-cache command.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

This was causing random segfaults, because use_packed_git() got
confused by random garbage there.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

The fsck-cache complains if objects referred to by files in .git/refs/
or objects stored in files under .git/objects/??/ are not found as
stand-alone SHA1 files (i.e.  found in alternate object pools
GIT_ALTERNATE_OBJECT_DIRECTORIES or packed archives stored under
.git/objects/pack).

Although this is a good semantics to maintain consistency of a single
.git/objects directory as a self contained set of objects, it sometimes
is useful to consider it is OK as long as these "outside" objects are
available.

This commit introduces a new flag, --standalone, to git-fsck-cache.
When it is not specified, connectivity checks and .git/refs pointer
checks are taught that it is OK when expected objects do not exist under
.git/objects/?? hierarchy but are available from an packed archive or in
an alternate object pool.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

This also adds a header with a signature, version info, and the number
of objects to the pack file.  It also encodes the file length and type
more efficiently.

This also suppresses creation of the index file.

We'll use this soon to write pack-files to stdout.

I'd stupidly only tested the non-branch-name version.

(And teach sha1_file and unpack-object know how to unpack them too, of
course)

We use sha1_object_info() now, and getting size is also trivial.

I admit that this is more of "because we can" not "because I see
immediate need for it", though.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

When trying to find out the type of the object, there is no need
to uncompress the whole object.  Just use sha1_object_info().
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

The initial one was not doing enough to figure things out
without uncompressing too much.  It also fixes a potential
segfault resulting from missing use_packed_git() call.

We would need to introduce unuse_packed_git() call and do proper
use counting to figure out when it is safe to unmap, but
currently we do not unmap packed file yet.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

t5300-pack-object.sh test 8 expected to have to create the "pack"
directory itself, and was unhappy when it already existed.

Now, there's still a misfeature there, which is that when you
create a new object, it doesn't check whether that object
already exists in the pack-file, so you'll end up with a few
recent objects that you really don't need (notably tree
objects), and this patch fixes it.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

This replaces sha1sum(1) with sum(1) in t/t1002.  GNU sum(1) runs in
"BSD compatibility" mode by default, and not all systems have GNU
coreutils.  On any system without GNU coreutils (or sha1sum) t1002 will
fail.  This patch should make t1002 complete successfully everywhere
that sum(1) runs.

I've tested this on Darwin and Linux; it works on both platforms.
Signed-off-by: NMark Allen <mrallen1@yahoo.com>
Acked-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Now there are 257 of them (256 numeric ones, and the new "pack" directory)

Don't make the user have to mkdir it just because he's excited about the
new object pack functionality, do it for him.

This would create broken pack archives for anything nontrivial.