Feedback:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7617#note_18763033

This refactors repository caching so it's possible to selectively
refresh certain caches, instead of just expiring and refreshing
everything.

To allow this the various methods that were cached (e.g. "tag_count" and
"readme") use a similar pattern that makes expiring and refreshing
their data much easier.

In this new setup caches are refreshed as follows:

1. After a commit (but before running ProjectCacheWorker) we expire some
   basic caches such as the commit count and repository size.

2. ProjectCacheWorker will recalculate the commit count, repository
   size, then refresh a specific set of caches based on the list of
   files changed in a push payload.

This requires a bunch of changes to the various methods that may be
cached. For one, data should not be cached if a branch used or the
entire repository does not exist. To prevent all these methods from
handling this manually this is taken care of in
Repository#cache_method_output. Some methods still manually check for
the existence of a repository but this result is also cached.

With selective flushing implemented ProjectCacheWorker no longer uses an
exclusive lease for all of its work. Instead this worker only uses a
lease to limit the number of times the repository size is updated as
this is a fairly expensive operation.

Also allow merge request to be merged with skipped pipeline and the
"only allow merge when pipeline is green" feature enabled
Signed-off-by: NRémy Coutable <remy@rymai.me>

Fixes #24677

Initializing Rugged objects is way too expensive just to check if a
repository exists. Even though we cache this data once in a while we
have to refresh this. On GitLab.com we have seen Repository#exists?
taking up to _1 minute_ to complete in the absolute worst case, though
usually it sits around a second or so.

Using File.exist? to instead check if $GIT_DIR/refs exists is a much
faster way of checking if a repository was initialized properly.

This moves the logic of detecting special repository files (e.g. a
README or a Koding configuration file) to a single class:
Gitlab::FileDetector. Moving this logic into a single place allows this
to be re-used more easily.

This commit also changes Repository#gitlab_ci_yaml so that its cached
similar to other data (e.g. the Koding configuration file).

Closes #23150

We need to handle annotated tags that are created via GitLab UI.
Annotated tags have their own SHA. We have to pass this SHA to
post-receive hook to mirror what happens when someone creates
an annotated tag in their local repository and pushes it via
command line.
In order to obtain tag SHA we first have to create it. This is
a bit confusing because we create the tag before executing
pre-hooks, but there is no way to create a tag outside the
repository. If pre-hooks fail we have to clean up after ourselves.

This also updates _some_ specs to use these new methods, just to serve
as an example for others going forward, but by no means is this
exhaustive.

Original implementations at !5992 and !6012.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/20944

This would fix long standing failures running tests on
my development machine, which set `Gitlab.config.gitlab.host`
to another host because it's not my local computer. Now I
finally cannot withstand it and decided to fix them once and
for all.

This prevents leakage of project names on an endpoint which is
unauthenticated and thus open to the world.

rather than using SHA for the pipeline link.

Signed-off-by: NRémy Coutable <remy@rymai.me>

gitlab-shell v3.6.6 would give project paths like so:

* namespace/project

gitlab-shell v4.0.0 can give project paths like so:

* /namespace1/namespace2/project
* /namespace/project
* /path/to/repository/storage/namespace1/namespace2/project
* /path/to/repository/storage/namespace/project