By passing commit data to this worker we remove the need for querying
the Git repository for every job. This in turn reduces the time spent
processing each job.

The migration included migrates jobs from the old format to the new
format. For this to work properly it requires downtime as otherwise
workers may start producing errors until they're using a newer version
of the worker code.

when we care only about the number of commits

We do not have to instantiate all objects in this case.

The implicit interface of project services states that the "execute"
method is meant to be called when project hooks are executed.
Currently JiraService does not support any project events even though
JiraService#supported_events says that "commit" and "merge_request"
are supported. They are only used to render correct options in
JIRA configuration screen, but they are not supported.
Because of that, this commit makes "execute" method a no-op.

Replace issue access checks with use of IssuableFinder

Split from !2024 to partially solve https://gitlab.com/gitlab-org/gitlab-ce/issues/23867

## Which fixes are in this MR?

⚠ - Potentially untested  
💣 - No test coverage  
🚥 - Test coverage of some sort exists (a test failed when error raised)  
🚦 - Test coverage of return value (a test failed when nil used)  
✅ - Permissions check tested

### Issue lookup with access check

Using `visible_to_user` likely makes these security issues too. See [Code smells](#code-smells).

- [x] 🚦 app/finders/notes_finder.rb:15 [`visible_to_user`]
- [x] 🚥 app/views/layouts/nav/_project.html.haml:73 [`visible_to_user`] [`.count`]
- [x] ✅ app/services/merge_requests/build_service.rb:84 [`issue.try(:confidential?)`]
- [x] ✅ lib/api/issues.rb:112 [`visible_to_user`]
  - CHANGELOG: Prevented API returning issues set to 'Only team members' to everyone
- [x] ✅ lib/api/helpers.rb:126 [`can?(current_user, :read_issue, issue)`] Maybe here too?
- [x] ✅ lib/gitlab/search_results.rb:53 [`visible_to_user`]

### Previous discussions
- [ ] https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2024/diffs#b2ff264eddf9819d7693c14ae213d941494fe2b3_128_126
- [ ] https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2024/diffs#7b6375270d22f880bdcb085e47b519b426a5c6c7_87_87

See merge request !2031

Fix missing access checks on issue lookup using IssuableFinder

Split from !2024 to partially solve https://gitlab.com/gitlab-org/gitlab-ce/issues/23867

⚠ - Potentially untested
💣 - No test coverage
🚥 - Test coverage of some sort exists (a test failed when error raised)
🚦 - Test coverage of return value (a test failed when nil used)
✅ - Permissions check tested

- [x] ✅ app/controllers/projects/branches_controller.rb:39
  - `before_action :authorize_push_code!` helpes limit/prevent exploitation. Always checks for reporter access so fine with
    confidential issues, issues only visible to team, etc.
- [x] 🚥 app/models/cycle_analytics/summary.rb:9 [`.count`]
- [x] ✅ app/controllers/projects/todos_controller.rb:19

- [x] Potential double render in app/controllers/projects/todos_controller.rb

- https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2024/diffs#cedccb227af9bfdf88802767cb58d43c2b977439_24_24

See merge request !2030

We only know the tag SHA after we create the tag.
This means that we pass a different value to the hooks that happen before
creating the tag, and a different value to the hooks that happen after
creating the tag.

This is not an ideal situation, but it is a trade-off we decided to
make. For discussion of the alternatives please refer to
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7700#note_18982873

"pre-receive" and "update" hooks always get the SHA of the commit
that the tag points to. "post-receive" gets the tag SHA if it is
an annotated tag or the commit SHA if it is an lightweight tag.
Currently we always create annotated tags if UI is used.

This reverts commit ae51774b.

* Added keyword arguments to truncated_diff_lines method to allow for using highlighting or not (html templates vs. text)
* Tweaked templates for consistency and format appropriateness

Previously the `truncated_diff_lines` method for outputting a discussion diff took in already highlighted lines, which meant it wasn't reuseable for truncating ANY lines. In the way it was used, it also meant that for any email truncation, the whole diff was being highlighted before being truncated, meaning wasted time highlighting lines that wouldn't even be used (granted, they were being memoized, so perhaps this wasn't that great of an issue). I refactored truncation away from highlighting, in order to truncate formatted diffs for text templates in email, using `>`s to designate each line, but otherwise retaining the parsing already done to create `diff_lines`.

Additionally, while notes on merge requests or commits had already been tested, there was no existing test for notes on a diff on an MR or commit. Added mailer tests for such, and a unit test for truncating diff lines.

With events no longer being cached this is no longer needed.

When I proposed using serializable transactions I was hoping we would be
able to refresh data of individual users concurrently. Unfortunately
upon closer inspection it was revealed this was not the case. This could
result in a lot of queries failing due to serialization errors,
overloading the database in the process (given enough workers trying to
update the target table).

To work around this we're now using a Redis lease that is cancelled upon
completion. This ensures we can update the data of different users
concurrently without overloading the database.

The code will try to obtain the lease until it succeeds, waiting at
least 1 second between retries. This is necessary as we may otherwise
end up _not_ updating the data which is not an option.

Signed-off-by: NDmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

Closes #23938

Feedback:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7508#note_18794681

Feedback:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7508#note_18794547

Feedback:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7617#note_18763033

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7508#note_18755739

Feedback:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7508#note_18748231

This refactors repository caching so it's possible to selectively
refresh certain caches, instead of just expiring and refreshing
everything.

To allow this the various methods that were cached (e.g. "tag_count" and
"readme") use a similar pattern that makes expiring and refreshing
their data much easier.

In this new setup caches are refreshed as follows:

1. After a commit (but before running ProjectCacheWorker) we expire some
   basic caches such as the commit count and repository size.

2. ProjectCacheWorker will recalculate the commit count, repository
   size, then refresh a specific set of caches based on the list of
   files changed in a push payload.

This requires a bunch of changes to the various methods that may be
cached. For one, data should not be cached if a branch used or the
entire repository does not exist. To prevent all these methods from
handling this manually this is taken care of in
Repository#cache_method_output. Some methods still manually check for
the existence of a repository but this result is also cached.

With selective flushing implemented ProjectCacheWorker no longer uses an
exclusive lease for all of its work. Instead this worker only uses a
lease to limit the number of times the repository size is updated as
this is a fairly expensive operation.

Also allow merge request to be merged with skipped pipeline and the
"only allow merge when pipeline is green" feature enabled
Signed-off-by: NRémy Coutable <remy@rymai.me>

Fixes #24677

Initializing Rugged objects is way too expensive just to check if a
repository exists. Even though we cache this data once in a while we
have to refresh this. On GitLab.com we have seen Repository#exists?
taking up to _1 minute_ to complete in the absolute worst case, though
usually it sits around a second or so.

Using File.exist? to instead check if $GIT_DIR/refs exists is a much
faster way of checking if a repository was initialized properly.

This moves the logic of detecting special repository files (e.g. a
README or a Koding configuration file) to a single class:
Gitlab::FileDetector. Moving this logic into a single place allows this
to be re-used more easily.

This commit also changes Repository#gitlab_ci_yaml so that its cached
similar to other data (e.g. the Koding configuration file).

Feedback:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7508#note_18730091

Closes #23150