I have reviewed all occurrences of mmap() in git and fixed three types
of errors/defects:

1) The result is not checked.
2) The file descriptor is closed if mmap() succeeds, but not when it
fails.
3) Various casts applied to -1 are used instead of MAP_FAILED, which is
specifically defined to check mmap() return value.

[jc: This is a second round of Pavel's patch.  He fixed up the problem
that close() potentially clobbering the errno from mmap, which
the first round had.]
Signed-off-by: NPavel Roskin <proski@gnu.org>
Signed-off-by: NJunio C Hamano <junkio@cox.net>

This reverses the order of object lookup, to check pack index first and
then go to the filesystem to find .git/objects/??/ hierarchy.

When most of the objects are packed, this saves quite many stat() calls
and negative dcache entries; while the price this approach has to pay is
negligible, even when most of the objects are outside pack, because
checking pack index file is quite cheap.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Very irritating.  But "snprintf()" wants "char *", and zlib wants
"unsigned char *".

Remove map_sha1_file(), now unused.
Signed-off-by: NDaniel Barkalow <barkalow@iabervon.org>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Add write_sha1_to_fd(), which writes an object to a file descriptor. This
includes support for unpacking it and recompressing it.
Signed-off-by: NDaniel Barkalow <barkalow@iabervon.org>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

This patch makes the first half of write_sha1_file() and
index_fd() externally visible, to allow callers to compute the
object ID without actually storing it in the object database.

[JC demangled the whitespaces himself because he liked the patch
 so much, and reworked the interface to index_fd() slightly,
 taking suggestion from Linus and of his own.]
Signed-off-by: NBryan Larsen <bryan.larsen@gmail.com>
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

I'm bored with doing it by hand all the time.

The function write_one_ref() is passed the list of refs received
from the other end, which was obtained by directory traversal
under $GIT_DIR/refs; this can contain paths other than what
git-init-db prepares and would fail to clone when there is
such.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

The function calls opendir() without a matching closedir().
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

GIT_DIR=.  ends up being what some of the pack senders use, and we
sometimes messed up when cleaning up the path, ie a ".//HEAD" was
cleaned up into "/HEAD", not "HEAD" like it should be.

We should do some other cleanup, and probably also verify that symlinks
don't point to outside the git area.

"git_path()" returns a static pathname pointer into the git directory
using a printf-like format specifier.

"head_ref()" works like "for_each_ref()", except for just the HEAD.

We'll want this for incremental packing.

Mainly making a lot of local functions and variables be marked "static",
but there was a "zero as NULL" warning in there too.

A couple of bogus "might be used undefined" warnings are avoided
by moving the initializations unnecessarily early.

This implements show_pack_info() function used in verify-pack
command when -v flag is used to obtain something like
unpack-objects used to give when it was first written.

It shows the following for each non-deltified object found in
the pack:

    SHA1 type size offset

For deltified objects, it shows this instead:

    SHA1 type size offset depth base_sha1

In order to get the output in the order that appear in the pack
file for debugging purposes, you can do this:

 $ git-verify-pack -v packfile | sort -n -k 4,4
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Nico pointed out that having verify_pack.c and verify-pack.c was
confusing.  Rename verify_pack.c to pack-check.c as suggested,
and enhances the verification done quite a bit.

 - Built-in sha1_file unpacking knows that a base object of a
   deltified object _must_ be in the same pack, and takes
   advantage of that fact.

 - Earlier verify-pack command only checked the SHA1 sum for the
   entire pack file and did not look into its contents.  It now
   checks everything idx file claims to have unpacks correctly.

 - It now has a hook to give more detailed information for
   objects contained in the pack under -v flag.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

This teaches packed_delta_info() that it only needs to look at
the type of the base object to figure out both type and size of
a deltified object.  This saves quite a many calls to inflate()
when dealing with a deep delta chain.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Given a list of <pack>.idx files, this command validates the
index file and the corresponding .pack file for consistency.

This patch also uses the same validation mechanism in fsck-cache
when the --full flag is used.

During normal operation, sha1_file.c verifies that a given .idx
file matches the .pack file by comparing the SHA1 checksum
stored in .idx file and .pack file as a minimum sanity check.
We may further want to check the pack signature and version when
we map the pack, but that would be a separate patch.

Earlier, errors to map a pack file was not flagged fatal but led
to a random fatal error later.  This version explicitly die()s
when such an error is detected.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

If we prefer 0 as maxsize for diff_delta() to say "unlimited", let's be
consistent about it.

This patch also fixes type mismatch in a call to get_delta_hdr_size()
from packed_delta_info().
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

This is a wrap-up patch including all the cleanups I've done to the
delta code and its usage.  The most important change is the
factorization of the delta header handling code.
Signed-off-by: NNicolas Pitre <nico@cam.org>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Pointed out by Junio.

This makes it match the new delta encoding, and admittedly makes the
code easier to follow.

This also updates the PACK file version to 2, since this (and the delta
encoding change in the previous commit) are incompatible with the old
format.

The commands git-fsck-cache and probably git-*-pull needs to have a way
to enumerate objects contained in packed GIT archives and alternate
object pools.  This commit exposes the data structure used to keep track
of them from sha1_file.c, and adds a couple of accessor interface
functions for use by the enhanced git-fsck-cache command.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

This was causing random segfaults, because use_packed_git() got
confused by random garbage there.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

This also adds a header with a signature, version info, and the number
of objects to the pack file.  It also encodes the file length and type
more efficiently.

(And teach sha1_file and unpack-object know how to unpack them too, of
course)

The initial one was not doing enough to figure things out
without uncompressing too much.  It also fixes a potential
segfault resulting from missing use_packed_git() call.

We would need to introduce unuse_packed_git() call and do proper
use counting to figure out when it is safe to unmap, but
currently we do not unmap packed file yet.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Now, there's still a misfeature there, which is that when you
create a new object, it doesn't check whether that object
already exists in the pack-file, so you'll end up with a few
recent objects that you really don't need (notably tree
objects), and this patch fixes it.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

GIT_OBJECT_DIRECTORY and GIT_ALTERNATE_OBJECT_DIRECTORIES can
have the "pack" subdirectory that houses "packed GIT" files
produced by git-pack-objects (e.g. .git/objects/pack/foo.pack
and .git/objects/pack/foo.idx; always store them as pairs).  The
following functions in sha1_file.c can then read object contents
from such packed file:

 - sha1_object_info()
 - has_sha1_file()
 - read_sha1_file()
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

This lets us eliminate one use of map_sha1_file() outside
sha1_file.c, to bring us one step closer to the packed GIT.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Packed delta files created by git-pack-objects seems to be the
way to go, and existing "delta" object handling code has exposed
the object representation details to too many places.  Remove it
while we refactor code to come up with a proper interface in
sha1_file.c.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Here is a patch that fixes several gcc4 warnings about different signedness,
all between char and unsigned char. I tried to keep the patch minimal
so resertod to casts in three places.
Signed-off-by: NMika Kukkonen <mikukkon@iki.fi>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Make 'sha1' parameters const where possible
Signed-off-by: NJason McMullan <jason.mcmullan@timesys.com>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

This patch adds code to read a hash out of a specified file under
{GIT_DIR}/refs/, and to write such files atomically and optionally with an
compare and lock.
Signed-off-by: NDaniel Barkalow <barkalow@iabervon.org>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

This adds sha1_file_size() helper function and uses it in the
rename/copy similarity estimator.  The helper function handles
deltified object as well.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

When a remote repository is deltified, we need to get the
objects that a deltified object we want to obtain is based upon.
The initial parts of each retrieved SHA1 file is inflated and
inspected to see if it is deltified, and its base object is
asked from the remote side when it is.  Since this partial
inflation and inspection has a small performance hit, it can
optionally be skipped by giving -d flag to git-*-pull commands.
This flag should be used only when the remote repository is
known to have no deltified objects.

Rsync transport does not have this problem since it fetches
everything the remote side has.
Signed-off-by: NJunio C Hamano <junkio@cox.net>
Signed-off-by: NLinus Torvalds <torvalds@osdl.org>

Make a separate helper for parsing the header of an object file
(really carefully) and for unpacking the rest. This means that
anybody who uses the "unpack_sha1_header()" interface can easily
look at the header and decide to unpack the rest too, without
doing any extra work.

It's for people who aren't necessarily interested in the whole
unpacked file, but do want to know the header information (size,
type, etc..)

For example, the delta code can use this to figure out whether
an object is already a delta object, and what it is a delta
against, without actually bothering to unpack all of the actual
data in the delta.

This makes them easier to parse without having to worry about
running off the end, and allows us to treat commits as normal
strings.

Add <limits.h> to the include files handled by "cache.h", and remove
extraneous #include directives from various .c files. The rule is that
"cache.h" gets all the basic stuff, so that we'll have as few system
dependencies as possible.