This reverts commit e00fb2bdc2090e9cabeb1eb35a2672a882cc96e9.

# Conflicts:
#	.rubocop.yml
#	.rubocop_todo.yml
#	lib/gitlab/ci/config/entry/global.rb
#	lib/gitlab/ci/config/entry/jobs.rb
#	spec/lib/gitlab/ci/config/entry/factory_spec.rb
#	spec/lib/gitlab/ci/config/entry/global_spec.rb
#	spec/lib/gitlab/ci/config/entry/job_spec.rb
#	spec/lib/gitlab/ci/status/build/factory_spec.rb
#	spec/lib/gitlab/incoming_email_spec.rb

This reverts commit cb10b725c8929b8b4460f89c9d96c773af39ba6b.

Backport changes from the EE-only squash implementation, which would
otherwise conflict when merge CE into EE.

<https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1024>

In repositories such as https://github.com/git/git.git, annotated tags can
point to blobs, not necessarily to commits. `Repository` attempts to
return the tags in the order of the commit date, but if a commit is not
available the previous implementation would error due to a `nil` target.
This change modifies the code to use the current time if a commit
is not associated with the given tag.

Closes #27228

Signed-off-by: NRémy Coutable <remy@rymai.me>

Feedback:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7237#note_19747856

more consistent across different methodst

So we still commit outside the hooks, and only
update ref inside the hooks. There are only two
exceptions:

* Whenever it's adding a tag. We can't add a tag
  without committing, unfortunately. See !7700
* Whenever source project is in another repository.
  We'll need to fetch ref otherwise commits can't be made.

See the whole discussion starting from:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7237#note_19210942

This method already uses the cached method Repository#branch_count so
there's no point in also caching has_visible_content?.

Fixes gitlab-org/gitlab-ce#25278

Add back check if we're losing commits in a merge.

git operation inside GitHooksService. Feedback:

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7237#note_19210942

TODO: Fix tests for update_branch_with_hooks

We only know the tag SHA after we create the tag.
This means that we pass a different value to the hooks that happen before
creating the tag, and a different value to the hooks that happen after
creating the tag.

This is not an ideal situation, but it is a trade-off we decided to
make. For discussion of the alternatives please refer to
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7700#note_18982873

"pre-receive" and "update" hooks always get the SHA of the commit
that the tag points to. "post-receive" gets the tag SHA if it is
an annotated tag or the commit SHA if it is an lightweight tag.
Currently we always create annotated tags if UI is used.

This reverts commit ae51774b.

This refactors repository caching so it's possible to selectively
refresh certain caches, instead of just expiring and refreshing
everything.

To allow this the various methods that were cached (e.g. "tag_count" and
"readme") use a similar pattern that makes expiring and refreshing
their data much easier.

In this new setup caches are refreshed as follows:

1. After a commit (but before running ProjectCacheWorker) we expire some
   basic caches such as the commit count and repository size.

2. ProjectCacheWorker will recalculate the commit count, repository
   size, then refresh a specific set of caches based on the list of
   files changed in a push payload.

This requires a bunch of changes to the various methods that may be
cached. For one, data should not be cached if a branch used or the
entire repository does not exist. To prevent all these methods from
handling this manually this is taken care of in
Repository#cache_method_output. Some methods still manually check for
the existence of a repository but this result is also cached.

With selective flushing implemented ProjectCacheWorker no longer uses an
exclusive lease for all of its work. Instead this worker only uses a
lease to limit the number of times the repository size is updated as
this is a fairly expensive operation.

Initializing Rugged objects is way too expensive just to check if a
repository exists. Even though we cache this data once in a while we
have to refresh this. On GitLab.com we have seen Repository#exists?
taking up to _1 minute_ to complete in the absolute worst case, though
usually it sits around a second or so.

Using File.exist? to instead check if $GIT_DIR/refs exists is a much
faster way of checking if a repository was initialized properly.

This moves the logic of detecting special repository files (e.g. a
README or a Koding configuration file) to a single class:
Gitlab::FileDetector. Moving this logic into a single place allows this
to be re-used more easily.

This commit also changes Repository#gitlab_ci_yaml so that its cached
similar to other data (e.g. the Koding configuration file).

We need to handle annotated tags that are created via GitLab UI.
Annotated tags have their own SHA. We have to pass this SHA to
post-receive hook to mirror what happens when someone creates
an annotated tag in their local repository and pushes it via
command line.
In order to obtain tag SHA we first have to create it. This is
a bit confusing because we create the tag before executing
pre-hooks, but there is no way to create a tag outside the
repository. If pre-hooks fail we have to clean up after ourselves.

gitlab-shell v3.6.6 would give project paths like so:

* namespace/project

gitlab-shell v4.0.0 can give project paths like so:

* /namespace1/namespace2/project
* /namespace/project
* /path/to/repository/storage/namespace1/namespace2/project
* /path/to/repository/storage/namespace/project