When a project is forked, the new repository used to be a deep copy of everything
stored on disk by leveraging `git clone`. This works well, and makes isolation
between repository easy. However, the clone is at the start 100% the same as the
origin repository. And in the case of the objects in the object directory, this
is almost always going to be a lot of duplication.

Object Pools are a way to create a third repository that essentially only exists
for its 'objects' subdirectory. This third repository's object directory will be
set as alternate location for objects. This means that in the case an object is
missing in the local repository, git will look in another location. This other
location is the object pool repository.

When Git performs garbage collection, it's smart enough to check the
alternate location. When objects are duplicated, it will allow git to
throw one copy away. This copy is on the local repository, where to pool
remains as is.

These pools have an origin location, which for now will always be a
repository that itself is not a fork. When the root of a fork network is
forked by a user, the fork still clones the full repository. Async, the
pool repository will be created.

Either one of these processes can be done earlier than the other. To
handle this race condition, the Join ObjectPool operation is
idempotent. Given its idempotent, we can schedule it twice, with the
same effect.

To accommodate the holding of state two migrations have been added.
1. Added a state column to the pool_repositories column. This column is
managed by the state machine, allowing for hooks on transitions.
2. pool_repositories now has a source_project_id. This column in
convenient to have for multiple reasons: it has a unique index allowing
the database to handle race conditions when creating a new record. Also,
it's nice to know who the host is. As that's a short link to the fork
networks root.

Object pools are only available for public project, which use hashed
storage and when forking from the root of the fork network. (That is,
the project being forked from itself isn't a fork)

In this commit message I use both ObjectPool and Pool repositories,
which are alike, but different from each other. ObjectPool refers to
whatever is on the disk stored and managed by Gitaly. PoolRepository is
the record in the database.

If a commit has two parents, one is direct ancestor, and one is not,
and the order of `commits` is in such fashion that the non-ancestor
side is visited first, the commit would be determined as non-ancestor,
when in fact it can be.

Therefore we should first determine all direct ancestors
prior to analyzing.

Analyze new commits graph to determine each commit's merge commit.

Fix "merged with [commit]" info for merge requests being merged
automatically by other actions.

Allow analyzing upto the relevant commit

Brings back 1e8f1de0 reverted in !23644

Closes #52342

See merge request gitlab-org/gitlab-ce!23436

This reverts commit 1e8f1de0, reversing
changes made to 62d97112.

This reverts commit 3560b119.

This changes `correlation_id` to be `correlation-id` when passed via jobs

The Correlation ID is taken or generated from received X-Request-ID.
Then it is being passed to all executed services (sidekiq workers
or gitaly calls).

The Correlation ID is logged in all structured logs as `correlation_id`.

* removed 100 limit on file search results because we
  load all results anyway
* expensive processing (parsing match content, utf encoding)
  is done only for selected page in paginated output

An unknown public GPG key will result in a GPGME::Error thrown from gpg,
which would cause an Error 500 on the signatures endpoint.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/54729

Signed-off-by: NRémy Coutable <remy@rymai.me>

The API permits path traversal characters like '../' to be passed down
to the template finder. Detect these requests and cause them to fail
with a 500 response code.

This reverts commit 22954f22

LfsToken::HMACToken#token_valid?() will be examined and if false, look
in redis via LfsToken::LegacyRedisDeviseToken#token_valid?().

AES-256-GCM cipher mode requires a key that is exactly 32 bytes long.
We already handle the case when the key is too long, by truncating, but
the key can also be too short in some installations. Switching to a key
that is always exactly the right length (by virtue of right-padding
ASCII 0 characters) allows encryption to proceed, without breaking
backward compatibility.

When the key is too short, encryption fails with an `ArgumentError`,
causing the web hooks functionality to be unusable. As a result, zero
rows can exist with values encrypted with the too-short key.

When the key is too long, it is silently truncated. In this case, the
key is unchanged, so values encrypted with the new too-long key will
still be successfully decrypted.

- Rename ordered_group_clusters_for_project ->
ancestor_clusters_for_clusterable
- Improve name of order option. It makes much more sense to have `hierarchy_order: :asc`
and `hierarchy_order: :desc`
- Allow ancestor_clusters_for_clusterable for group
- Re-use code already present in Project

Look for matching clusters starting from the closest ancestor, then go
up the ancestor tree.

Then use Ruby to get clusters for each group in order. Not that
efficient, considering we will doing up to `NUMBER_OF_ANCESTORS_ALLOWED`
number of queries, but it's a finite number

Explicitly order query by depth

This allows us to control ordering explicitly and also to reverse the
order which is useful to allow us to be consistent with
Clusters::Cluster.on_environment (EE) which does reverse ordering.

Puts querying group clusters behind Feature Flag. Just in case we have
issues with performance, we can easily disable this

If the service fails mid-point, then we should be able to re-run this
service. So, detect presence of any previously created Kubernetes
resource and update or create accordingly.

Fix specs accordingly. In the case of finalize_creation_service_spec.rb,
I decided to stub out the async worker rather than maintaining
individual stubs for various kubeclient calls for that worker.

Currently, if a job does not have only/except policies, the policy is considered as an unspecified state, and therefore the job is executed regardless of how it's executed and which branch/tags are targetted.
Ideally, this should be specified as only: ['branches', 'tags'], as it indicates that unspecified policy jobs are meant to run on any git references.

This reverts merge request !23165

If the service fails mid-point, then we should be able to re-run this
service. So, detect presence of any previously created Kubernetes
resource and update or create accordingly.

Fix specs accordingly. In the case of finalize_creation_service_spec.rb,
I decided to stub out the async worker rather than maintaining
individual stubs for various kubeclient calls for that worker.

Also add test cases for group clusters

These changes make the code more reusable, testable, and most
importantly, overrideable.

The numbers in
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22650#note_117645395
are quite promising with that target.

This improves readability quite a bit.