These delta pointers always point to elements in the objects[] array
in packing_data struct. We can only hold maximum 4G of those objects
because the array size in nr_objects is uint32_t. We could use
uint32_t indexes to address these elements instead of pointers. On
64-bit architecture (8 bytes per pointer) this would save 4 bytes per
pointer.

Convert these delta pointers to indexes. Since we need to handle NULL
pointers as well, the index is shifted by one [1].

[1] This means we can only index 2^32-2 objects even though nr_objects
    could contain 2^32-1 objects. It should not be a problem in
    practice because when we grow objects[], nr_alloc would probably
    blow up long before nr_objects hits the wall.
Signed-off-by: NNguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Instead of using 8 bytes (on 64 bit arch) to store a pointer to a
pack. Use an index instead since the number of packs should be
relatively small.

This limits the number of packs we can handle to 1k. Since we can't be
sure people can never run into the situation where they have more than
1k pack files. Provide a fall back route for it.

If we find out they have too many packs, the new in_pack_by_idx[]
array (which has at most 1k elements) will not be used. Instead we
allocate in_pack[] array that holds nr_objects elements. This is
similar to how the optional in_pack_pos field is handled.

The new simple test is just to make sure the too-many-packs code path
is at least executed. The true test is running

    make test GIT_TEST_FULL_IN_PACK_ARRAY=1

to take advantage of other special case tests.
Signed-off-by: NNguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

This field is only need for pack-bitmap, which is an optional
feature. Move it to a separate array that is only allocated when
pack-bitmap is used (like objects[], it is not freed, since we need it
until the end of the process)
Signed-off-by: NNguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Because of struct packing from now on we can only handle max depth
4095 (or even lower when new booleans are added in this struct). This
should be ok since long delta chain will cause significant slow down
anyway.
Signed-off-by: NNguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Signed-off-by: NNguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

An extra field type_valid is added to carry the equivalent of OBJ_BAD
in the original "type" field. in_pack_type always contains a valid
type so we only need 3 bits for it.

A note about accepting OBJ_NONE as "valid" type. The function
read_object_list_from_stdin() can pass this value [1] and it
eventually calls create_object_entry() where current code skip setting
"type" field if the incoming type is zero. This does not have any bad
side effects because "type" field should be memset()'d anyway.

But since we also need to set type_valid now, skipping oe_set_type()
leaves type_valid zero/false, which will make oe_type() return
OBJ_BAD, not OBJ_NONE anymore. Apparently we do care about OBJ_NONE in
prepare_pack(). This switch from OBJ_NONE to OBJ_BAD may trigger

    fatal: unable to get type of object ...

Accepting OBJ_NONE [2] does sound wrong, but this is how it is has
been for a very long time and I haven't time to dig in further.

[1] See 5c49c116 (pack-objects: better check_object() performances -
    2007-04-16)

[2] 21666f1a (convert object type handling from a string to a number
    - 2007-02-26)
Signed-off-by: NNguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

The role of this comment block becomes more important after we shuffle
fields around to shrink this struct. It will be much harder to see what
field is related to what.
Signed-off-by: NNguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

While at there, document about this special mode when running the test
suite.
Signed-off-by: NNguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

* 'svn/authors-prog-2' of git://bogomips.org/git-svn:
  git-svn: allow empty email-address using authors-prog and authors-file
  git-svn: search --authors-prog in PATH too

This reverts commit e4bb62fa, reversing
changes made to 468165c1.

The topic appears to inflict severe regression in renaming merges,
even though the promise of it was that it would improve them.

We do not yet know which exact change in the topic was wrong, but in
the meantime, let's play it safe and revert it out of 'master'
before real Git-using projects are harmed.
Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Signed-off-by: NJunio C Hamano <gitster@pobox.com>

When credential helper exits very quickly without reading its
input, it used to cause Git to die with SIGPIPE, which has been
fixed.

* eb/cred-helper-ignore-sigpipe:
  credential: ignore SIGPIPE when writing to credential helpers

When built with more recent cURL, GIT_SSL_VERSION can now specify
"tlsv1.3" as its value.

* lv/tls-1.3:
  http: allow use of TLS 1.3

Test cleanup.

* pk/test-avoid-pipe-hiding-exit-status:
  test: avoid pipes in git related commands for test

"git submodule status" misbehaved on a submodule that has been
removed from the working tree.

* rs/status-with-removed-submodule:
  submodule: check for NULL return of get_submodule_ref_store()

Small test-helper programs have been consolidated into a single
binary.

* nd/combined-test-helper: (36 commits)
  t/helper: merge test-write-cache into test-tool
  t/helper: merge test-wildmatch into test-tool
  t/helper: merge test-urlmatch-normalization into test-tool
  t/helper: merge test-subprocess into test-tool
  t/helper: merge test-submodule-config into test-tool
  t/helper: merge test-string-list into test-tool
  t/helper: merge test-strcmp-offset into test-tool
  t/helper: merge test-sigchain into test-tool
  t/helper: merge test-sha1-array into test-tool
  t/helper: merge test-scrap-cache-tree into test-tool
  t/helper: merge test-run-command into test-tool
  t/helper: merge test-revision-walking into test-tool
  t/helper: merge test-regex into test-tool
  t/helper: merge test-ref-store into test-tool
  t/helper: merge test-read-cache into test-tool
  t/helper: merge test-prio-queue into test-tool
  t/helper: merge test-path-utils into test-tool
  t/helper: merge test-online-cpus into test-tool
  t/helper: merge test-mktemp into test-tool
  t/helper: merge (unused) test-mergesort into test-tool
  ...

Refactoring of the internal global data structure continues.

* sb/packfiles-in-repository:
  packfile: keep prepare_packed_git() private
  packfile: allow find_pack_entry to handle arbitrary repositories
  packfile: add repository argument to find_pack_entry
  packfile: allow reprepare_packed_git to handle arbitrary repositories
  packfile: allow prepare_packed_git to handle arbitrary repositories
  packfile: allow prepare_packed_git_one to handle arbitrary repositories
  packfile: add repository argument to reprepare_packed_git
  packfile: add repository argument to prepare_packed_git
  packfile: add repository argument to prepare_packed_git_one
  packfile: allow install_packed_git to handle arbitrary repositories
  packfile: allow rearrange_packed_git to handle arbitrary repositories
  packfile: allow prepare_packed_git_mru to handle arbitrary repositories

Refactoring the internal global data structure to make it possible
to open multiple repositories, work with and then close them.

Rerolled by Duy on top of a separate preliminary clean-up topic.
The resulting structure of the topics looked very sensible.

* sb/object-store: (27 commits)
  sha1_file: allow sha1_loose_object_info to handle arbitrary repositories
  sha1_file: allow map_sha1_file to handle arbitrary repositories
  sha1_file: allow map_sha1_file_1 to handle arbitrary repositories
  sha1_file: allow open_sha1_file to handle arbitrary repositories
  sha1_file: allow stat_sha1_file to handle arbitrary repositories
  sha1_file: allow sha1_file_name to handle arbitrary repositories
  sha1_file: add repository argument to sha1_loose_object_info
  sha1_file: add repository argument to map_sha1_file
  sha1_file: add repository argument to map_sha1_file_1
  sha1_file: add repository argument to open_sha1_file
  sha1_file: add repository argument to stat_sha1_file
  sha1_file: add repository argument to sha1_file_name
  sha1_file: allow prepare_alt_odb to handle arbitrary repositories
  sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
  sha1_file: add repository argument to prepare_alt_odb
  sha1_file: add repository argument to link_alt_odb_entries
  sha1_file: add repository argument to read_info_alternates
  sha1_file: add repository argument to link_alt_odb_entry
  sha1_file: add raw_object_store argument to alt_odb_usable
  pack: move approximate object count to object store
  ...

Test helper update.

* jc/test-must-be-empty:
  test_must_be_empty: simplify file existence check

Perf-test update.

* cc/perf-aggregate-sort:
  perf/aggregate: add --sort-by=regression option
  perf/aggregate: add display_dir()

Doc updates.

* ab/doc-hash-brokenness:
  doc hash-function-transition: clarify what SHAttered means
  doc hash-function-transition: clarify how older gits die on NewHash

Tests that rely on the exact hardcoded values of object names have
been updated in preparation for hash function migration.

* bc/hash-independent-tests:
  t2107: abstract away SHA-1-specific constants
  t2101: abstract away SHA-1-specific constants
  t2101: modernize test style
  t2020: abstract away SHA-1 specific constants
  t1507: abstract away SHA-1-specific constants
  t1411: abstract away SHA-1-specific constants
  t1405: sort reflog entries in a hash-independent way
  t1300: abstract away SHA-1-specific constants
  t1304: abstract away SHA-1-specific constants
  t1011: abstract away SHA-1-specific constants

* ab/drop-contrib-examples:
  Remove contrib/examples/*

Signed-off-by: NJunio C Hamano <gitster@pobox.com>

Code clean-up.

* ys/bisect-object-id-missing-conversion-fix:
  bisect: use oid_to_hex() for converting object_id hashes to hex strings

when refs that do not point at committish are given, "git
filter-branch" gave a misleading error messages.  This has been
corrected.

* yk/filter-branch-non-committish-refs:
  filter-branch: fix errors caused by refs that point at non-committish

Code cleanup.

* nd/trace-with-env:
  run-command: use strbuf_addstr() for adding a string to a strbuf

The mechanism to use parse-options API to automate the command line
completion continues to get extended and polished.

* nd/parseopt-completion-more:
  completion: use __gitcomp_builtin in _git_cherry
  completion: use __gitcomp_builtin in _git_ls_tree
  completion: delete option-only completion commands
  completion: add --option completion for most builtin commands
  completion: factor out _git_xxx calling code
  completion: mention the oldest version we need to support
  git.c: add hidden option --list-parseopt-builtins
  git.c: move cmd_struct declaration up

Code to find the length to uniquely abbreviate object names based
on packfile content, which is a relatively recent addtion, has been
optimized to use the same fan-out table.

* ds/bsearch-hash:
  sha1_name: use bsearch_pack() in unique_in_pack()
  sha1_name: use bsearch_pack() for abbreviations
  packfile: define and use bsearch_pack()
  sha1_name: convert struct min_abbrev_data to object_id

Code clean-up.

* ws/rebase-p:
  rebase: remove merges_option and a blank line
  rebase: remove unused code paths from git_rebase__interactive__preserve_merges
  rebase: remove unused code paths from git_rebase__interactive
  rebase: add and use git_rebase__interactive__preserve_merges
  rebase: extract functions out of git_rebase__interactive
  rebase: reindent function git_rebase__interactive
  rebase: update invocation of rebase dot-sourced scripts
  rebase-interactive: simplify pick_on_preserving_merges

"diff-highlight" filter (in contrib/) learned to undertand "git log
--graph" output better.

* jk/diff-highlight-graph-fix:
  diff-highlight: detect --graph by indent
  diff-highlight: use flush() helper consistently
  diff-highlight: test graphs with --color
  diff-highlight: test interleaved parallel lines of history
  diff-highlight: prefer "echo" to "cat" in tests
  diff-highlight: use test_tick in graph test
  diff-highlight: correct test graph diagram

Code clean-up for the "repository" abstraction.

* nd/remove-ignore-env-field:
  repository.h: add comment and clarify repo_set_gitdir
  repository: delete ignore_env member
  sha1_file.c: move delayed getenv(altdb) back to setup_git_env()
  repository.c: delete dead functions
  repository.c: move env-related setup code back to environment.c
  repository: initialize the_repository in main()

"git tag --contains no-such-commit" gave a full list of options
after giving an error message.

* ps/contains-id-error-message:
  parse-options: do not show usage upon invalid option value

Signed-off-by: NJunio C Hamano <gitster@pobox.com>

"git stash push -u -- <pathspec>" gave an unnecessary and confusing
error message when there was no tracked files that match the
<pathspec>, which has been fixed.

* tg/stash-untracked-with-pathspec-fix:
  stash: drop superfluos pathspec parameter
  stash push -u: don't create empty stash
  stash push: avoid printing errors
  stash: fix nonsense pipeline

The way "git worktree prune" worked internally has been simplified,
by assuming how "git worktree move" moves an existing worktree to a
different place.

* nd/worktree-prune:
  worktree prune: improve prune logic when worktree is moved
  worktree: delete dead code
  gc.txt: more details about what gc does

Conversion from uchar[20] to struct object_id continues.

* bc/object-id: (36 commits)
  convert: convert to struct object_id
  sha1_file: introduce a constant for max header length
  Convert lookup_replace_object to struct object_id
  sha1_file: convert read_sha1_file to struct object_id
  sha1_file: convert read_object_with_reference to object_id
  tree-walk: convert tree entry functions to object_id
  streaming: convert istream internals to struct object_id
  tree-walk: convert get_tree_entry_follow_symlinks internals to object_id
  builtin/notes: convert static functions to object_id
  builtin/fmt-merge-msg: convert remaining code to object_id
  sha1_file: convert sha1_object_info* to object_id
  Convert remaining callers of sha1_object_info_extended to object_id
  packfile: convert unpack_entry to struct object_id
  sha1_file: convert retry_bad_packed_offset to struct object_id
  sha1_file: convert assert_sha1_type to object_id
  builtin/mktree: convert to struct object_id
  streaming: convert open_istream to use struct object_id
  sha1_file: convert check_sha1_signature to struct object_id
  sha1_file: convert read_loose_object to use struct object_id
  builtin/index-pack: convert struct ref_delta_entry to object_id
  ...

"git shortlog cruft" aborted with a BUG message when run outside a
Git repository.  The command has been taught to complain about
extra and unwanted arguments on its command line instead in such a
case.

* ma/shortlog-revparse:
  shortlog: disallow left-over arguments outside repo
  shortlog: add usage-string for stdin-reading
  git-shortlog.txt: reorder usages

The build procedure learned to optionally use symbolic links
(instead of hardlinks and copies) to install "git-foo" for built-in
commands, whose binaries are all identical.

* ab/install-symlinks:
  Makefile: optionally symlink libexec/git-core binaries to bin/git
  Makefile: add a gitexecdir_relative variable
  Makefile: fix broken bindir_relative variable

"git filter-branch" learned to use a different exit code to allow
the callers to tell the case where there was no new commits to
rewrite from other error cases.

* ml/filter-branch-no-op-error:
  filter-branch: return 2 when nothing to rewrite