"git pack-objects" needs to allocate tons of "struct object_entry"
while doing its work, and shrinking its size helps the performance
quite a bit.

* nd/pack-objects-pack-struct:
  ci: exercise the whole test suite with uncommon code in pack-objects
  pack-objects: reorder members to shrink struct object_entry
  pack-objects: shrink delta_size field in struct object_entry
  pack-objects: shrink size field in struct object_entry
  pack-objects: clarify the use of object_entry::size
  pack-objects: don't check size when the object is bad
  pack-objects: shrink z_delta_size field in struct object_entry
  pack-objects: refer to delta objects by index instead of pointer
  pack-objects: move in_pack out of struct object_entry
  pack-objects: move in_pack_pos out of struct object_entry
  pack-objects: use bitfield for object_entry::depth
  pack-objects: use bitfield for object_entry::dfs_state
  pack-objects: turn type and in_pack_type to bitfields
  pack-objects: a bit of document about struct object_entry
  read-cache.c: make $GIT_TEST_SPLIT_INDEX boolean

Rename detection logic in "diff" family that is used in "merge" has
learned to guess when all of x/a, x/b and x/c have moved to z/a,
z/b and z/c, it is likely that x/d added in the meantime would also
want to move to z/d by taking the hint that the entire directory
'x' moved to 'z'.  A bug causing dirty files involved in a rename
to be overwritten during merge has also been fixed as part of this
work.  Incidentally, this also avoids updating a file in the
working tree after a (non-trivial) merge whose result matches what
our side originally had.

* en/rename-directory-detection-reboot: (36 commits)
  merge-recursive: fix check for skipability of working tree updates
  merge-recursive: make "Auto-merging" comment show for other merges
  merge-recursive: fix remainder of was_dirty() to use original index
  merge-recursive: fix was_tracked() to quit lying with some renamed paths
  t6046: testcases checking whether updates can be skipped in a merge
  merge-recursive: avoid triggering add_cacheinfo error with dirty mod
  merge-recursive: move more is_dirty handling to merge_content
  merge-recursive: improve add_cacheinfo error handling
  merge-recursive: avoid spurious rename/rename conflict from dir renames
  directory rename detection: new testcases showcasing a pair of bugs
  merge-recursive: fix remaining directory rename + dirty overwrite cases
  merge-recursive: fix overwriting dirty files involved in renames
  merge-recursive: avoid clobbering untracked files with directory renames
  merge-recursive: apply necessary modifications for directory renames
  merge-recursive: when comparing files, don't include trees
  merge-recursive: check for file level conflicts then get new name
  merge-recursive: add computation of collisions due to dir rename & merging
  merge-recursive: check for directory level conflicts
  merge-recursive: add get_directory_renames()
  merge-recursive: make a helper function for cleanup for handle_renames
  ...

"git --no-pager cmd" did not have short-and-sweet single letter
option. Now it does.

* js/no-pager-shorthand:
  git: add -P as a short option for --no-pager

"git rebase -i" sometimes left intermediate "# This is a
combination of N commits" message meant for the human consumption
inside an editor in the final result in certain corner cases, which
has been fixed.

* js/rebase-i-clean-msg-after-fixup-continue:
  rebase --skip: clean up commit message after a failed fixup/squash
  sequencer: always commit without editing when asked for
  rebase -i: Handle "combination of <n> commits" with GETTEXT_POISON
  rebase -i: demonstrate bugs with fixup!/squash! commit messages

"git worktree add" learned to check out an existing branch.

* tg/worktree-add-existing-branch:
  worktree: teach "add" to check out existing branches
  worktree: factor out dwim_branch function
  worktree: improve message when creating a new worktree
  worktree: remove extra members from struct add_opts

The functionality of "$GIT_DIR/info/grafts" has been superseded by
the "refs/replace/" mechanism for some time now, but the internal
code had support for it in many places, which has been cleaned up
in order to drop support of the "grafts" mechanism.

* js/deprecate-grafts:
  Remove obsolete script to convert grafts to replace refs
  technical/shallow: describe why shallow cannot use replace refs
  technical/shallow: stop referring to grafts
  filter-branch: stop suggesting to use grafts
  Deprecate support for .git/info/grafts
  Add a test for `git replace --convert-graft-file`
  replace: introduce --convert-graft-file
  replace: prepare create_graft() for converting graft files wholesale
  replace: "libify" create_graft() and callees
  replace: avoid using die() to indicate a bug
  commit: Let the callback of for_each_mergetag return on error
  argv_array: offer to split a string by whitespace

Test debugging aid.

* js/test-unset-prereq:
  tests: introduce test_unset_prereq, for debugging

We learned that our source files with ".pl" and ".py" extensions
are Perl and Python files respectively and changes to them are
better viewed as such with appropriate diff drivers.

* ab/perl-python-attrs:
  .gitattributes: add a diff driver for Python
  .gitattributes: use the "perl" differ for Perl
  .gitattributes: add *.pl extension for Perl

The codepath around object-info API has been taught to take the
repository object (which in turn tells the API which object store
the objects are to be located).

* sb/oid-object-info:
  cache.h: allow oid_object_info to handle arbitrary repositories
  packfile: add repository argument to cache_or_unpack_entry
  packfile: add repository argument to unpack_entry
  packfile: add repository argument to read_object
  packfile: add repository argument to packed_object_info
  packfile: add repository argument to packed_to_object_type
  packfile: add repository argument to retry_bad_packed_offset
  cache.h: add repository argument to oid_object_info
  cache.h: add repository argument to oid_object_info_extended

Perf-test helper updates.

* cc/perf-aggregate-unknown-option:
  perf/aggregate: use Getopt::Long for option parsing

Dev support.

* en/git-debugger:
  Make running git under other debugger-like programs easy

The transport protocol v2 is getting updated further.

* bw/server-options:
  fetch: send server options when using protocol v2
  ls-remote: send server options when using protocol v2
  serve: introduce the server-option capability

"git gc" in a large repository takes a lot of time as it considers
to repack all objects into one pack by default.  The command has
been taught to pretend as if the largest existing packfile is
marked with ".keep" so that it is left untouched while objects in
other packs and loose ones are repacked.

* nd/repack-keep-pack:
  pack-objects: show some progress when counting kept objects
  gc --auto: exclude base pack if not enough mem to "repack -ad"
  gc: handle a corner case in gc.bigPackThreshold
  gc: add gc.bigPackThreshold config
  gc: add --keep-largest-pack option
  repack: add --keep-pack option
  t7700: have closing quote of a test at the beginning of line

The code has been taught to use the duplicated information stored
in the commit-graph file to learn the tree object name for a commit
to avoid opening and parsing the commit object when it makes sense
to do so.

* ds/lazy-load-trees:
  coccinelle: avoid wrong transformation suggestions from commit.cocci
  commit-graph: lazy-load trees for commits
  treewide: replace maybe_tree with accessor methods
  commit: create get_commit_tree() method
  treewide: rename tree to maybe_tree

The code did not propagate the terminal width to subprocesses via
COLUMNS environment variable, which it now does.  This caused
trouble to "git column" helper subprocess when "git tag --column=row"
tried to list the existing tags on a display with non-default width.

* nd/term-columns:
  column: fix off-by-one default width
  pager: set COLUMNS to term_columns()

Test update.

* sg/t7005-spaces-in-filenames-cleanup:
  t7005-editor: get rid of the SPACES_IN_FILENAMES prereq

Test fixes.

* sg/t5516-fixes:
  t5516-fetch-push: fix broken &&-chain
  t5516-fetch-push: fix 'push with dry-run' test

Test update.

* sg/t5310-jgit-bitmap-test:
  t5310-pack-bitmaps: make JGit tests work with GIT_TEST_SPLIT_INDEX

Doc update.

* nd/pack-format-doc:
  pack-format.txt: more details on pack file format

Doc update.

* jk/apply-p-doc:
  apply: clarify "-p" documentation

Doc update.

* ao/config-api-doc:
  doc: fix config API documentation about config_with_options

* bc/mailmap-self:
  mailmap: update brian m. carlson's email address

Hotfix.

* sb/object-store-replace:
  get_main_ref_store: BUG() when outside a repository
  object.c: clear replace map before freeing it
  replace-object.c: remove the_repository from prepare_replace_object
  object.c: free replace map in raw_object_store_clear

Hotfix.

* hn/sort-ls-remote:
  t5512: run git fetch inside test

If we don't have a repository, then we can't initialize the
ref store.  Prior to 64a74161 (refs: store the main ref
store inside the repository struct, 2018-04-11), we'd try to
access get_git_dir(), and outside a repository that would
trigger a BUG(). After that commit, though, we directly use
the_repository->git_dir; if it's NULL we'll just segfault.

Let's catch this case and restore the BUG() behavior.
Obviously we don't ever want to hit this code, but a BUG()
is a lot more helpful than a segfault if we do.
Signed-off-by: NJeff King <peff@peff.net>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Signed-off-by: NStefan Beller <sbeller@google.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The last two tests 'editor with a space' and 'core.editor with a
space' in 't7005-editor.sh' need the SPACES_IN_FILENAMES prereq to
ensure that they are only run on filesystems that allow, well, spaces
in filenames.  However, we have been putting a space in the name of
the trash directory for just over a decade now, so we wouldn't be able
to run any of our tests on such a filesystem in the first place.

This prereq is therefore unnecessary, remove it.
Signed-off-by: NSZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Do the preparatory fetch inside the test of ls-remote --symref to avoid
cluttering the test output and to be able to catch unexpected fetch
failures.
Signed-off-by: NRene Scharfe <l.s.r@web.de>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

By default we want to fill the whole screen if possible, but we do not
want to use up _all_ terminal columns because the last character is
going hit the border, push the cursor over and wrap. Keep it at
default value zero, which will make print_columns() set the width at
term_columns() - 1.

This affects the test in t7004 because effective column width before
was 40 but now 39 so we need to compensate it by one or the output at
39 columns has a different layout.
Signed-off-by: NNguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

After we invoke the pager, our stdout goes to a pipe, not the
terminal, meaning we can no longer use an ioctl to get the
terminal width. For that reason, ad6c3739 (pager: find out
the terminal width before spawning the pager, 2012-02-12)
started caching the terminal width.

But that cache is only an in-process variable. Any programs
we spawn will also not be able to run that ioctl, but won't
have access to our cache. They'll end up falling back to our
80-column default.

You can see the problem with:

  git tag --column=row

Since git-tag spawns a pager these days, its spawned
git-column helper will see neither the terminal on stdout
nor a useful COLUMNS value (assuming you do not export it
from your shell already). And you'll end up with 80-column
output in the pager, regardless of your terminal size.

We can fix this by setting COLUMNS right before spawning the
pager. That fixes this case, as well as any more complicated
ones (e.g., a paged program spawns another script which then
generates columnized output).
Signed-off-by: NJeff King <peff@peff.net>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The current document mentions OBJ_* constants without their actual
values. A git developer would know these are from cache.h but that's
not very friendly to a person who wants to read this file to implement
a pack file parser.

Similarly, the deltified representation is not documented at all (the
"document" is basically patch-delta.c). Translate that C code to
English with a bit more about what ofs-delta and ref-delta mean.
Signed-off-by: NNguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

We're not really removing slashes, but slash-separated path
components. Let's make that more clear.
Reported-by: Nkelly elton <its.the.doc@gmail.com>
Signed-off-by: NJeff King <peff@peff.net>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

b2dc968e (t5516: refactor oddball tests, 2008-11-07) accidentaly
broke the &&-chain in the test 'push does not update local refs on
failure', but since it was in a subshell, chain-lint couldn't notice
it.
Signed-off-by: NSZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

In a while-at-it cleanup replacing a 'cd dir && <...> && cd ..' with a
subshell, commit 28391a80 (receive-pack: allow deletion of corrupt
refs, 2007-11-29) also moved the assignment of the $old_commit
variable to that subshell.  This variable, however, is used outside of
that subshell as a parameter of check_push_result(), to check that a
ref still points to the commit where it is supposed to.  With the
variable remaining unset outside the subshell check_push_result()
doesn't perform that check at all.

Use 'git -C <dir> cmd...', so we don't need to change directory, and
thus don't need the subshell either when setting $old_commit.

Furthermore, change check_push_result() to require at least three
parameters (the repo, the oid, and at least one ref), so it will catch
similar issues earlier should they ever arise.
Signed-off-by: NSZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The two JGit tests 'we can read jgit bitmaps' and 'jgit can read our
bitmaps' in 't5310-pack-bitmaps.sh' fail when run with
GIT_TEST_SPLIT_INDEX=YesPlease.  Both tests create a clone of the test
repository to check bitmap interoperability with JGit.  With split
indexes enabled the index in the clone repositories contains the
'link' extension, which JGit doesn't support and, consequently, an
exception aborts it:

  <...>
  org.eclipse.jgit.api.errors.JGitInternalException: DIRC extension 'link' not supported by this version.
          at org.eclipse.jgit.dircache.DirCache.readFrom(DirCache.java:562)
  <...>

Since testing bitmaps doesn't need a worktree in the first place,
let's just create bare clones for the two JGit tests, so the cloned
won't have an index, and these two tests can be executed even with
split index enabled.
Signed-off-by: NSZEDER Gábor <szeder.dev@gmail.com>
Reviewed-by: NJeff King <peff@peff.net>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

In commit dc8441fd ("config: don't implicitly use gitdir or commondir",
2017-06-14) the function git_config_with_options was renamed to
config_with_options to better reflect the fact that it does not access
the git global config or the repo config by default.

However Documentation/technical/api-config.txt still refers to the
previous name, fix that.

While at it also update the documentation about the extra parameters,
because they too changed since the initial definition.
Signed-off-by: NAntonio Ospite <ao2@ao2.it>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

This was missed in 5982da9d (replace-object: allow
prepare_replace_object to handle arbitrary repositories, 2018-04-11)

Technically the code works correctly as the replace_map is the same
size in different repositories, however it is hard to read. So convert
the code to the familiar pattern of dereferencing the pointer that we
assign in the sizeof itself.
Signed-off-by: NStefan Beller <sbeller@google.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The replace map for objects was missed to free in the object store in
the conversion of 174774cd (Merge branch 'sb/object-store-replace',
2018-05-08)
Signed-off-by: NStefan Beller <sbeller@google.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The can-working-tree-updates-be-skipped check has had a long and blemished
history.  The update can be skipped iff:
  a) The merge is clean
  b) The merge matches what was in HEAD (content, mode, pathname)
  c) The target path is usable (i.e. not involved in D/F conflict)

Traditionally, we split b into parts:
  b1) The merged result matches the content and mode found in HEAD
  b2) The merged target path existed in HEAD

Steps a & b1 are easy to check; we have always gotten those right.  While
it is easy to overlook step c, this was fixed seven years ago with commit
4ab9a157 ("merge_content(): Check whether D/F conflicts are still
present", 2010-09-20).  merge-recursive didn't have a readily available
way to directly check step b2, so various approximations were used:

  * In commit b2c8c0a7 ("merge-recursive: When we detect we can skip
    an update, actually skip it", 2011-02-28), it was noted that although
    the code claimed it was skipping the update, it did not actually skip
    the update.  The code was made to skip it, but used lstat(path, ...)
    as an approximation to path-was-tracked-in-index-before-merge.

  * In commit 5b448b85 ("merge-recursive: When we detect we can skip
    an update, actually skip it", 2011-08-11), the problem with using
    lstat was noted.  It was changed to the approximation
       path2 && strcmp(path, path2)
    which is also wrong.  !path2 || strcmp(path, path2) would have been
    better, but would have fallen short with directory renames.

  * In c5b761fb ("merge-recursive: ensure we write updates for
    directory-renamed file", 2018-02-14), the problem with the previous
    approximation was noted and changed to
       was_tracked(path)
    That looks close to what we were trying to answer, but was_tracked()
    as implemented at the time should have been named is_tracked(); it
    returned something different than what we were looking for.

  * To make matters more complex, fixing was_tracked() isn't sufficient
    because the splitting of b into b1 and b2 is wrong.  Consider the
    following merge with a rename/add conflict:
       side A: modify foo, add unrelated bar
       side B: rename foo->bar (but don't modify the mode or contents)
    In this case, the three-way merge of original foo, A's foo, and B's
    bar will result in a desired pathname of bar with the same
    mode/contents that A had for foo.  Thus, A had the right mode and
    contents for the file, and it had the right pathname present (namely,
    bar), but the bar that was present was unrelated to the contents, so
    the working tree update was not skippable.

Fix this by introducing a new function:
   was_tracked_and_matches(o, path, &mfi.oid, mfi.mode)
and use it to directly check for condition b.
Signed-off-by: NElijah Newren <newren@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Previously, merge_content() would print "Auto-merging" whenever the final
content and mode aren't already available from HEAD.  There are a few
problems with this:

  1) There are other code paths doing merges that should probably have the
     same message printed, in particular rename/rename(2to1) which cannot
     call into the normal rename logic.

  2) If both sides of the merge have modifications, then a content merge
     is needed.  It may turn out that the end result matches one of the
     sides (because the other only had a subset of the same changes), but
     the merge was still needed.  Currently, the message will not print in
     that case, though it seems like it should.

Move the printing of this message to merge_file_1() in order to address
both issues.
Signed-off-by: NElijah Newren <newren@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>