Helped-by: NJeff King <peff@peff.net>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Make map_sha1_file(), parse_sha1_header() and unpack_sha1_header()
available to the streaming read API by exporting them via cache.h header
file.
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Helped-by: NJeff King <peff@peff.net>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

One typical use of a large binary file is to hold a sparse on-disk hash
table with a lot of holes. Help preserving the holes with lseek().
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Signed-off-by: NJunio C Hamano <gitster@pobox.com>

When the output to a path does not have to be converted, we can read from
the object database from the streaming API and write to the file in the
working tree, without having to hold everything in the memory.

The ident, auto- and safe- crlf conversions inherently require you to read
the whole thing before deciding what to do, so while it is technically
possible to support them by using a buffer of an unbound size or rewinding
and reading the stream twice, it is less practical than the traditional
"read the whole thing in core and convert" approach.

Adding streaming filters for the other conversions on top of this should
be doable by tweaking the can_bypass_conversion() function (it should be
renamed to can_filter_stream() when it happens). Then the streaming API
can be extended to wrap the git_istream streaming_write_entry() opens on
the underlying object in another git_istream that reads from it, filters
what is read, and let the streaming_write_entry() read the filtered
result. But that is outside the scope of this series.
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Given an object name, use open_istream() to get a git_istream handle
that you can read_istream() from as if you are using read(2) to read
the contents of the object, and close it with close_istream() when
you are done.

Currently, we do not do anything fancy--it just calls read_sha1_file()
and keeps the contents in memory as a whole, and carve it out as you
request with read_istream().
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

In the write-out codepath, a block of code determines what file in the
working tree to write to, and opens an output file descriptor to it.

After writing the contents out to the file, another block of code runs
fstat() on the file descriptor when appropriate.

Separate these blocks out to open_output_fd() and fstat_output()
helper functions.
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

This function is used to read and skip over the per-object header
in a packfile.
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

An object found in the delta-base cache is not guaranteed to
stay there, but we know it came from a pack and it is likely
to give us a quick access if we read_sha1_file() it right now,
which is a piece of useful information.
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The original interface for sha1_object_info() takes an object name and
gives back a type and its size (the latter is given only when it was
asked).  The new interface wraps its implementation and exposes a bit
more pieces of information that the interface used to discard, namely:

 - where the object is stored (loose? cached? packed?)
 - if packed, where in which packfile?
Signed-off-by: NJunio C Hamano <gitster@pobox.com>
---

 * In the earlier round, this used u.pack.delta to record the length of
   the delta chain, but the caller is not necessarily interested in the
   length of the delta chain per-se, but may only want to know if it is a
   delta against another object or is stored as a deflated data. Calling
   packed_object_info_detail() involves walking the reverse index chain to
   compute the store size of the object and is unnecessarily expensive.

   We could resurrect the code if a new caller wants to know, but I doubt
   it.

Instead return an integer that can be given to typename() if
the caller wants a string, just like everybody else does.
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

* jc/convert:
  convert: make it harder to screw up adding a conversion attribute
  convert: make it safer to add conversion attributes
  convert: give saner names to crlf/eol variables, types and functions
  convert: rename the "eol" global variable to "core_eol"

* jc/bigfile:
  Bigfile: teach "git add" to send a large file straight to a pack
  index_fd(): split into two helper functions
  index_fd(): turn write_object and format_check arguments into one flag

* jc/replacing:
  read_sha1_file(): allow selective bypassing of replacement mechanism
  inline lookup_replace_object() calls
  read_sha1_file(): get rid of read_sha1_file_repl() madness
  t6050: make sure we test not just commit replacement
  Declare lookup_replace_object() in cache.h, not in commit.h

Signed-off-by: NJunio C Hamano <gitster@pobox.com>

* maint:
  Update draft release notes to 1.7.5.2
  git_open_noatime(): drop unused parameter
  sha1_file: typofix

Signed-off-by: NJunio C Hamano <gitster@pobox.com>

* cn/format-patch-quiet:
  format-patch: document --quiet option
  format-patch: don't pass on the --quiet flag

* jm/mergetool-submodules:
  mergetool: Teach about submodules

* jk/format-patch-quote-special-in-from:
  pretty: quote rfc822 specials in email addresses

* vh/git-svn-doc:
  git-svn.txt: small typeface improvements
  git-svn.txt: move option descriptions
  git-svn.txt: fix usage of --add-author-from

Since commit c7934306 (Limit file descriptors used by packs, 2011-02-28),
the extra parameter added in f2e872aa (Work around EMFILE when there are
too many pack files, 2010-11-01) is not used anymore.

Remove it.
Signed-off-by: NJunio C Hamano <gitster@pobox.com>
Acked-by: NShawn O. Pearce <spearce@spearce.org>

The number zero is spelled "zero", not "zer0".
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The way "object replacement" mechanism was tucked to the read_sha1_file()
interface was suboptimal in a couple of ways:

 - Callers that want it to die with useful diagnosis upon seeing a corrupt
   object does not have a way to say that they do not want any object
   replacement.

 - Callers who do not want it to die but want to handle the errors
   themselves are told to arrange to call read_object(), but the function
   does not use the replacement mechanism, and also it is a file scope
   static function that not many callers can call to begin with.

This adds a read_sha1_file_extended() that takes a set of flags; the
callers of read_sha1_file() passes a flag READ_SHA1_FILE_REPLACE to ask
for object replacement mechanism to kick in.

Later, we could add another flag bit to tell the function to return an
error instead of dying and then remove the misguided "call read_object()
yourself".
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

In a repository without object replacement, lookup_replace_object() should
be a no-op. Check the flag "read_replace_refs" on the side of the caller,
and bypess a function call when we know we are not dealing with replacement.

Also, even when we are set up to replace objects, if we do not find any
replacement defined, flip that flag off to avoid function call overhead
for all the later object accesses.

As this change the semantics of the flag from "do we need read the
replacement definition?" to "do we need to check with the lookup table?"
the flag needs to be renamed later to something saner, e.g. "use_replace",
when the codebase is calmer, but not now.
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Most callers want to silently get a replacement object, and they do not
care what the real name of the replacement object is.  Worse yet, no sane
interface to return the underlying object without replacement is provided.

Remove the function and make only the few callers that want the name of
the replacement object find it themselves.
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The replacement mechanism should affect all types of objects not
just commits, so make sure it deals with at least a blob.
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The declaration is misplaced as the replace API is supposed to affect
not just commits, but all types of objects.
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

* maint:
  add, merge, diff: do not use strcasecmp to compare config variable names

The config machinery already makes section and variable names
lowercase when parsing them, so using strcasecmp for comparison just
feels wasteful.  No noticeable change intended.
Noticed-by: NJay Soffian <jaysoffian@gmail.com>
Signed-off-by: NJonathan Nieder <jrnieder@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

When adding a new content to the repository, we have always slurped
the blob in its entirety in-core first, and computed the object name
and compressed it into a loose object file.  Handling large binary
files (e.g.  video and audio asset for games) has been problematic
because of this design.

At the middle level of "git add" callchain is an internal API
index_fd() that takes an open file descriptor to read from the
working tree file being added with its size. Teach it to call out to
fast-import when adding a large blob.

The write-out codepath in entry.c::write_entry() should be taught to
stream, instead of reading everything in core. This should not be so
hard to implement, especially if we limit ourselves only to loose
object files and non-delta representation in packfiles.
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Signed-off-by: NJunio C Hamano <gitster@pobox.com>

* bf/commit-template-no-cleanup:
  Do not strip empty lines / trailing spaces from a commit message template

* jc/t1506-shell-param-expansion-gotcha:
  t1507: avoid "${parameter<op>'word'}" inside double-quotes

* rr/rerere-libify-clear-gc:
  rerere: libify rerere_clear() and rerere_gc()

* js/maint-send-pack-stateless-rpc-deadlock-fix:
  send-pack: unbreak push over stateless rpc
  send-pack: avoid deadlock when pack-object dies early

* jh/dirstat-lines:
  Mark dirstat error messages for translation
  Improve error handling when parsing dirstat parameters
  New --dirstat=lines mode, doing dirstat analysis based on diffstat
  Allow specifying --dirstat cut-off percentage as a floating point number
  Add config variable for specifying default --dirstat behavior
  Refactor --dirstat parsing; deprecate --cumulative and --dirstat-by-file
  Make --dirstat=0 output directories that contribute < 0.1% of changes
  Add several testcases for --dirstat and friends

* jc/fix-add-u-unmerged:
  Fix "add -u" that sometimes fails to resolve unmerged paths

* jn/setup-revisions-glob-and-friends-passthru:
  revisions: allow --glob and friends in parse_options-enabled commands
  revisions: split out handle_revision_pseudo_opt function

* cn/log-parse-opt:
  log: convert to parse-options

* maint:
  Prepare for 1.7.5.2
  t5400: Fix a couple of typos

Conflicts:
	RelNotes