the `ci_pipelines.sha` column is not the same type than the
`merge_request_diff_commits.sha` column (varchar, bytea)

This throttles the number of UPDATE queries that can be triggered by
calling "touch" on a Note, Issue, or MergeRequest. For Note objects we
also take care of updating the associated "noteable" relation in a
smarter way than Rails does by default.

If a merge request was created with a branch name that also matched a tag name,
we'd generate a comparison to or from the tag respectively, rather than the
branch. Merging would still use the branch, of course.

To avoid this, ensure that when we get the branch heads, we prepend the
reference prefix for branches, which will ensure that we generate the correct
comparison.

The st_commits and st_diffs columns on merge_request_diffs historically held the
YAML-serialised data for a merge request diff, in a variety of formats.

Since 9.5, these have been migrated in the background to two new tables:
merge_request_diff_commits and merge_request_diff_files. That has the advantage
that we can actually query the data (for instance, to find out how many commits
we've stored), and that it can't be in a variety of formats, but must match the
new schema.

This is the final step of that journey, where we drop those columns and remove
all references to them. This is a breaking change to the importer, because we
can no longer import diffs created in the old format, and we cannot guarantee
the export will be in the new format unless it was generated after this commit.

Compared to the merge_request_diff association:

1. It's simpler to query. The query uses a foreign key to the
   merge_request_diffs table, so no ordering is necessary.
2. It's faster for preloading. The merge_request_diff association has to load
   every diff for the MRs in the set, then discard all but the most recent for
   each. This association means that Rails can just query for N diffs from N
   MRs.
3. It's more complicated to update. This is a bidirectional foreign key, so we
   need to update two tables when adding a diff record. This also means we need
   to handle this as a special case when importing a GitLab project.

There is some juggling with this association in the merge request model:

* `MergeRequest#latest_merge_request_diff` is _always_ the latest diff.
* `MergeRequest#merge_request_diff` reuses
  `MergeRequest#latest_merge_request_diff` unless:
    * Arguments are passed. These are typically to force-reload the association.
    * It doesn't exist. That means we might be trying to implicitly create a
      diff. This only seems to happen in specs.
    * The association is already loaded. This is important for the reasons
      explained in the comment, which I'll reiterate here: if we a) load a
      non-latest diff, then b) get its `merge_request`, then c) get that MR's
      `merge_request_diff`, we should get the diff we loaded in c), even though
      that's not the latest diff.

Basically, `MergeRequest#merge_request_diff` is the latest diff in most cases,
but not quite all.

When we consider 'all' pipelines for MRs, we now mean:

1. The last 10,000 commits (unordered).
2. From the last 100 MR versions (newest first).

This seems to fix the MRs that time out on GitLab.com.

Use Commit#notes and Note.for_commit_id when possible to make sure we use all the indexes available to us

reword the changelog
remove dead code in the specs

also, I refactored the MergeRequest#fetch_ref method to express
the side-effect that this method has.

  MergeRequest#fetch_ref -> MergeRequest#fetch_ref!
  Repository#fetch_source_branch -> Repository#fetch_source_branch!

For MRs with many thousands of commits, `SELECT DISTINCT(sha)` will be very
slow.

What we can't do to fix this:

1. Add an index. Postgres won't use it for DISTINCT without a lot of ceremony.
2. Do the `uniq` in Ruby. That can still be very slow with hundreds of
   thousands of commits.
3. Use a subquery. We haven't removed the `st_commits` column yet, but we will
   soon.

Until 3 is available to us, we can just do 2, but also add a limit clause. There
is no ordering, so this may return different results, but our goal with these
MRs is just to get them to load, so it's not a huge deal.

Signed-off-by: NRémy Coutable <remy@rymai.me>

Signed-off-by: NRémy Coutable <remy@rymai.me>

Signed-off-by: NRémy Coutable <remy@rymai.me>

In GitLab EE, a GitLab instance can be read-only (e.g. when it's a Geo
secondary node). But in GitLab CE it also might be useful to have the
"read-only" idea around. So port it back to GitLab CE.

Also having the principle of read-only in GitLab CE would hopefully
lead to less errors introduced, doing write operations when there
aren't allowed for read-only calls.

Closes gitlab-org/gitlab-ce#37534.

MergeRequest#create_merge_request_diff and MergeRequest#reload_diff are
the only places where we generate a new MR diff so that's where we
should fetch the ref.

This also ensures that the ref is not fetched when we call
merge_request.merge_request_diffs.create in
Github::Import#fetch_pull_requests.
Signed-off-by: NRémy Coutable <remy@rymai.me>

In this particular case the use of UNION ALL leads to a better query
plan compared to using 1 big query that uses an OR statement to combine
different data sources.

See https://gitlab.com/gitlab-org/gitlab-ce/issues/38508 for more
information.

This ensures the open issues/MR count caches are refreshed properly when
creating new issues or MRs. This MR also includes a change to the cache
keys to ensure all caches are rebuilt on the fly.

This particular problem was not caught in the test suite due to a null
cache being used, resulting in all calls that would use a cache using
the underlying data directly. In production the code would fail because
a newly saved record returns an empty hash in #changes meaning checks
such as `state_changed? || confidential_changed?` would return false for
new rows, thus never updating the counters.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/38061