Given a list of <pack>.idx files, this command validates the
index file and the corresponding .pack file for consistency.

This patch also uses the same validation mechanism in fsck-cache
when the --full flag is used.

During normal operation, sha1_file.c verifies that a given .idx
file matches the .pack file by comparing the SHA1 checksum
stored in .idx file and .pack file as a minimum sanity check.
We may further want to check the pack signature and version when
we map the pack, but that would be a separate patch.

Earlier, errors to map a pack file was not flagged fatal but led
to a random fatal error later.  This version explicitly die()s
when such an error is detected.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

If we prefer 0 as maxsize for diff_delta() to say "unlimited", let's be
consistent about it.

This patch also fixes type mismatch in a call to get_delta_hdr_size()
from packed_delta_info().
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

This is a wrap-up patch including all the cleanups I've done to the
delta code and its usage.  The most important change is the
factorization of the delta header handling code.
Signed-off-by: NNicolas Pitre <nico@cam.org>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Pointed out by Junio.

This makes it match the new delta encoding, and admittedly makes the
code easier to follow.

This also updates the PACK file version to 2, since this (and the delta
encoding change in the previous commit) are incompatible with the old
format.

Since the delta data format is not tied to any actual git object
anymore, now is the time to add a small improvement to the delta data
header as it is been done for packed object header.  This patch allows
for reducing the delta header of about 2 bytes and makes for simpler
code.
Signed-off-by: NNicolas Pitre <nico@cam.org>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Deltas are useless by themselves and when you use them you need to get
to their base objects.  A base object should inherit recency from the
most recent deltified object that is based on it and that is what this
patch teaches git-pack-objects.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

It gets a bit more complicated to unpack in a streaming environment, but
here it is.  The rewrite is actually a lot cleaner in other ways, it's
just a bit more subtle.

The diff_delta() interface was extended to reject generating too big a
delta while we were working on the packed GIT archive format.

Take advantage of that when generating delta in the similarity estimator
used in diffcore-rename.c
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Standalone unpack-objects command was not adjusted for header length
encoding change when dealing with deltified entry.  This fixes it.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

It's just easier to let git help out with the merge than it is to try to
fix up the diffs.

The fsck-cache complains if objects referred to by files in .git/refs/
or objects stored in files under .git/objects/??/ are not found as
stand-alone SHA1 files (i.e.  found in alternate object pools
GIT_ALTERNATE_OBJECT_DIRECTORIES or packed archives stored under
.git/objects/pack).

Although this is a good semantics to maintain consistency of a single
.git/objects directory as a self contained set of objects, it sometimes
is useful to consider it is OK as long as these "outside" objects are
available.

This commit introduces a new flag, --standalone, to git-fsck-cache.
When it is not specified, connectivity checks and .git/refs pointer
checks are taught that it is OK when expected objects do not exist under
.git/objects/?? hierarchy but are available from an packed archive or in
an alternate object pool.

Another new flag, --full, makes git-fsck-cache to check not only the
current GIT_OBJECT_DIRECTORY but also objects found in alternate object
pools and packed GIT archives.a
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

The commands git-fsck-cache and probably git-*-pull needs to have a way
to enumerate objects contained in packed GIT archives and alternate
object pools.  This commit exposes the data structure used to keep track
of them from sha1_file.c, and adds a couple of accessor interface
functions for use by the enhanced git-fsck-cache command.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

This was causing random segfaults, because use_packed_git() got
confused by random garbage there.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

The fsck-cache complains if objects referred to by files in .git/refs/
or objects stored in files under .git/objects/??/ are not found as
stand-alone SHA1 files (i.e.  found in alternate object pools
GIT_ALTERNATE_OBJECT_DIRECTORIES or packed archives stored under
.git/objects/pack).

Although this is a good semantics to maintain consistency of a single
.git/objects directory as a self contained set of objects, it sometimes
is useful to consider it is OK as long as these "outside" objects are
available.

This commit introduces a new flag, --standalone, to git-fsck-cache.
When it is not specified, connectivity checks and .git/refs pointer
checks are taught that it is OK when expected objects do not exist under
.git/objects/?? hierarchy but are available from an packed archive or in
an alternate object pool.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

This also adds a header with a signature, version info, and the number
of objects to the pack file.  It also encodes the file length and type
more efficiently.

This also suppresses creation of the index file.

We'll use this soon to write pack-files to stdout.

I'd stupidly only tested the non-branch-name version.

(And teach sha1_file and unpack-object know how to unpack them too, of
course)

We use sha1_object_info() now, and getting size is also trivial.

I admit that this is more of "because we can" not "because I see
immediate need for it", though.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

When trying to find out the type of the object, there is no need
to uncompress the whole object.  Just use sha1_object_info().
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

The initial one was not doing enough to figure things out
without uncompressing too much.  It also fixes a potential
segfault resulting from missing use_packed_git() call.

We would need to introduce unuse_packed_git() call and do proper
use counting to figure out when it is safe to unmap, but
currently we do not unmap packed file yet.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

t5300-pack-object.sh test 8 expected to have to create the "pack"
directory itself, and was unhappy when it already existed.

Now, there's still a misfeature there, which is that when you
create a new object, it doesn't check whether that object
already exists in the pack-file, so you'll end up with a few
recent objects that you really don't need (notably tree
objects), and this patch fixes it.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

This replaces sha1sum(1) with sum(1) in t/t1002.  GNU sum(1) runs in
"BSD compatibility" mode by default, and not all systems have GNU
coreutils.  On any system without GNU coreutils (or sha1sum) t1002 will
fail.  This patch should make t1002 complete successfully everywhere
that sum(1) runs.

I've tested this on Darwin and Linux; it works on both platforms.
Signed-off-by: NMark Allen <mrallen1@yahoo.com>
Acked-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Now there are 257 of them (256 numeric ones, and the new "pack" directory)

Don't make the user have to mkdir it just because he's excited about the
new object pack functionality, do it for him.

This would create broken pack archives for anything nontrivial.

GIT_OBJECT_DIRECTORY and GIT_ALTERNATE_OBJECT_DIRECTORIES can
have the "pack" subdirectory that houses "packed GIT" files
produced by git-pack-objects (e.g. .git/objects/pack/foo.pack
and .git/objects/pack/foo.idx; always store them as pairs).  The
following functions in sha1_file.c can then read object contents
from such packed file:

 - sha1_object_info()
 - has_sha1_file()
 - read_sha1_file()
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

This lets us eliminate one use of map_sha1_file() outside
sha1_file.c, to bring us one step closer to the packed GIT.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Packed delta files created by git-pack-objects seems to be the
way to go, and existing "delta" object handling code has exposed
the object representation details to too many places.  Remove it
while we refactor code to come up with a proper interface in
sha1_file.c.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

In contrast to other plumbing tools, git-ssh-push only
allow a very restrictive form of commit-id filenames.
This patch removes this restriction.
Acked-by: NDaniel Barkalow <barkalow@iabervon.org>
Signed-off-by: NSven Verdoolaege <skimo@kotnet.org>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Sven Verdoolaege points out that I added the --verify option to
git-rev-parse exactly for things like this, but didn't update the
users.

If we have a very long commit message, and we end up getting a
bufferfull of data from git-rev-list that all belongs to one commit,
we ended up throwing away the data from a previous read that should
have been included.  The result was a error message about not being
able to parse the output of git-rev-list.

Also, if the git-rev-list output that we can't parse is long, only put
the first 80 chars in the error message.  Otherwise we end up with an
enormous error window.

Also, make the writing of the SHA1 as a end-header be conditional: not
every user will necessarily want to write the SHA1 to the file itself,
even though current users do (but we migh end up using the same helper
functions for the object files themselves, that don't do this).

This also makes the packed index file contain the SHA1 of the packed
data file at the end (just before its own SHA1).  That way you can
validate the pairing of the two if you want to.