Backport changes from the EE-only squash implementation, which would
otherwise conflict when merge CE into EE.

<https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1024>

In repositories such as https://github.com/git/git.git, annotated tags can
point to blobs, not necessarily to commits. `Repository` attempts to
return the tags in the order of the commit date, but if a commit is not
available the previous implementation would error due to a `nil` target.
This change modifies the code to use the current time if a commit
is not associated with the given tag.

Closes #27228

Feedback:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7237#note_21626953
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7237#note_21626952

Signed-off-by: NRémy Coutable <remy@rymai.me>

We merge repository checks inside it so we don't have to
check it on the call site, and we could also load the commit
for the caller. This greatly reduce code duplication.

Feedback:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7237#note_20572919

Feedback:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7237#note_19747244

It's very weird that source_commit.raw_commit and
rugged.branches[merge_request.target_branch].target
should be completely the same. I checked with ==
and other values which proved that both should be
the same, but still tests cannot pass for:
spec/services/merge_requests/refresh_service_spec.rb

I decided to give it up. We could just use SHA and
that works fine anyway.

Feedback:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7237#note_19746468

Feedback:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7237#note_19747456

Replace MR access checks with use of MergeRequestsFinder

Split from !2024 to partially solve https://gitlab.com/gitlab-org/gitlab-ce/issues/23867

⚠ - Potentially untested
💣 - No test coverage
🚥 - Test coverage of some sort exists (a test failed when error raised)
🚦 - Test coverage of return value (a test failed when nil used)
✅ - Permissions check tested

- [x] 💣  app/finders/notes_finder.rb:17
- [x] ⚠  app/views/layouts/nav/_project.html.haml:80 [`.count`]
- [x] 💣  app/controllers/concerns/creates_commit.rb:84
- [x] 🚥  app/controllers/projects/commits_controller.rb:24
- [x] 🚥  app/controllers/projects/compare_controller.rb:56
- [x] 🚦  app/controllers/projects/discussions_controller.rb:29
- [x] ✅  app/controllers/projects/todos_controller.rb:27
- [x] 🚦  app/models/commit.rb:268
- [x] ✅ lib/gitlab/search_results.rb:71

- [x] https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2024/diffs#d1c10892daedb4d4dd3d4b12b6d071091eea83df_267_266 Memoize ` merged_merge_request(current_user)`
- [x] https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2024/diffs#d1c10892daedb4d4dd3d4b12b6d071091eea83df_248_247 Expected side effect for `merged_merge_request!`, consider `skip_authorization: true`.
- [x] https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2024/diffs#d1c10892daedb4d4dd3d4b12b6d071091eea83df_269_269 Scary use  of unchecked `merged_merge_request?`

See merge request !2033

commits from the other repository. We'll cleanup
the tmp ref after we're done with our business.

more consistent across different methodst

by checking filename as well

have the branch existed upfront. That is,
`Rugged::Commit.create` rather than `Gitlab::Git::Blob.commit`
which the former doesn't need to have the branch but the latter
needs.

So we still commit outside the hooks, and only
update ref inside the hooks. There are only two
exceptions:

* Whenever it's adding a tag. We can't add a tag
  without committing, unfortunately. See !7700
* Whenever source project is in another repository.
  We'll need to fetch ref otherwise commits can't be made.

See the whole discussion starting from:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7237#note_19210942

Feedback:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7237/diffs#note_19210818

This method already uses the cached method Repository#branch_count so
there's no point in also caching has_visible_content?.

Fixes gitlab-org/gitlab-ce#25278

git operation inside GitHooksService. Feedback:

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7237#note_19210942

TODO: Fix tests for update_branch_with_hooks

We only know the tag SHA after we create the tag.
This means that we pass a different value to the hooks that happen before
creating the tag, and a different value to the hooks that happen after
creating the tag.

This is not an ideal situation, but it is a trade-off we decided to
make. For discussion of the alternatives please refer to
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7700#note_18982873

"pre-receive" and "update" hooks always get the SHA of the commit
that the tag points to. "post-receive" gets the tag SHA if it is
an annotated tag or the commit SHA if it is an lightweight tag.
Currently we always create annotated tags if UI is used.

This reverts commit ae51774b.

and keep it only called in update_branch_with_hooks.

This refactors repository caching so it's possible to selectively
refresh certain caches, instead of just expiring and refreshing
everything.

To allow this the various methods that were cached (e.g. "tag_count" and
"readme") use a similar pattern that makes expiring and refreshing
their data much easier.

In this new setup caches are refreshed as follows:

1. After a commit (but before running ProjectCacheWorker) we expire some
   basic caches such as the commit count and repository size.

2. ProjectCacheWorker will recalculate the commit count, repository
   size, then refresh a specific set of caches based on the list of
   files changed in a push payload.

This requires a bunch of changes to the various methods that may be
cached. For one, data should not be cached if a branch used or the
entire repository does not exist. To prevent all these methods from
handling this manually this is taken care of in
Repository#cache_method_output. Some methods still manually check for
the existence of a repository but this result is also cached.

With selective flushing implemented ProjectCacheWorker no longer uses an
exclusive lease for all of its work. Instead this worker only uses a
lease to limit the number of times the repository size is updated as
this is a fairly expensive operation.