The st_commits and st_diffs columns on merge_request_diffs historically held the
YAML-serialised data for a merge request diff, in a variety of formats.

Since 9.5, these have been migrated in the background to two new tables:
merge_request_diff_commits and merge_request_diff_files. That has the advantage
that we can actually query the data (for instance, to find out how many commits
we've stored), and that it can't be in a variety of formats, but must match the
new schema.

This is the final step of that journey, where we drop those columns and remove
all references to them. This is a breaking change to the importer, because we
can no longer import diffs created in the old format, and we cannot guarantee
the export will be in the new format unless it was generated after this commit.

Compared to the merge_request_diff association:

1. It's simpler to query. The query uses a foreign key to the
   merge_request_diffs table, so no ordering is necessary.
2. It's faster for preloading. The merge_request_diff association has to load
   every diff for the MRs in the set, then discard all but the most recent for
   each. This association means that Rails can just query for N diffs from N
   MRs.
3. It's more complicated to update. This is a bidirectional foreign key, so we
   need to update two tables when adding a diff record. This also means we need
   to handle this as a special case when importing a GitLab project.

There is some juggling with this association in the merge request model:

* `MergeRequest#latest_merge_request_diff` is _always_ the latest diff.
* `MergeRequest#merge_request_diff` reuses
  `MergeRequest#latest_merge_request_diff` unless:
    * Arguments are passed. These are typically to force-reload the association.
    * It doesn't exist. That means we might be trying to implicitly create a
      diff. This only seems to happen in specs.
    * The association is already loaded. This is important for the reasons
      explained in the comment, which I'll reiterate here: if we a) load a
      non-latest diff, then b) get its `merge_request`, then c) get that MR's
      `merge_request_diff`, we should get the diff we loaded in c), even though
      that's not the latest diff.

Basically, `MergeRequest#merge_request_diff` is the latest diff in most cases,
but not quite all.

This adds various foreign keys and indexes to the "merge_requests" table
as outlined in https://gitlab.com/gitlab-org/gitlab-ce/issues/31825.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/31825

This finishes the procedure for migrating events from the old format
into the new format. Code no longer uses the old setup and the database
tables used during the migration process are swapped, with the old table
being dropped.

While the database migration can be reversed this will 1) take a lot of
time as data has to be coped around 2) won't restore data in the
"events.data" column as we have no way of restoring this.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/37241

Remove superfluous lib: true, type: redis, service: true, models: true, services: true, no_db: true, api: true
Signed-off-by: NRémy Coutable <remy@rymai.me>

This is an ID-less table with just three columns: an association to the merge
request diff the commit belongs to, the relative order of the commit within the
merge request diff, and the commit SHA itself.

Previously we stored much more information about the commits, so that we could
display them even when they were deleted from the repo. Since 8.0, we ensure
that those commits are kept around for as long as the target repo itself is, so
we don't need to duplicate that data in the database.

This adds an ID-less table containing one row per file, per merge request
diff. It has a column for each attribute on Gitlab::Git::Diff that is serialised
currently, with the advantage that we can easily query the attributes of this
new table.

It does not migrate existing data, so we have fallback code when the legacy
st_diffs column is present instead. For a merge request diff to be valid, it
should have at most one of:

* Rows in this new table, with the correct merge_request_diff_id.
* A non-NULL st_diffs column.

It may have neither, if the diff is empty.

Signed-off-by: NRémy Coutable <remy@rymai.me>

https://gitlab.com/gitlab-org/gitlab-ce/issues/24007

* Regenerates tokens for all models that have them
* Remove variables, since they are basically just storing encrypted data
* Bumped version up to 0.1.6
* Updated related docs

Fix symlink vulnerability in Import/Export

Replaces https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2018 made by @james

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/23822

See merge request !2022
Signed-off-by: NRémy Coutable <remy@rymai.me>

Fixed all related specs and also changed the logic to handle edge cases. This includes exporting and exporting of group labels, which will get associated with the new group (if any) or they will become normal project labels otherwise.

Found other issues to do with not being able to import all labels at once in the beginning of the JSON - code was much simpler when we import all labels and milestones associated to a project first, then the associations will find the already created labels instead of creating them from the associations themselves.