The first attempt didn't migrate all rows on GitLab.com, due to a couple of
issues:

1. Some rows in merge_request_diffs had truly huge numbers of commits and diffs
   serialised - one in particular had 26,000 commits!
2. The jobs were sometimes on Sidekiq hosts with frequent OOM errors, leading to
   the job being lost.

The previous commit adds more logging, and a more robust insertion method. This
commit reschedules the jobs, with a generous pause between each.

This is used in EE for the storage migration, and we want to use this
in CE as well to be able to migrate projects to hashed_storage.

This is a composite index on (target_project_id, merge_commit_sha, id)
that allows queries such as the following to use a full backwards index
scan:

    SELECT "merge_requests".*
    FROM "merge_requests"
    WHERE "merge_requests"."deleted_at" IS NULL
    AND "merge_requests"."target_project_id" = 13083
    AND "merge_requests"."merge_commit_sha" = 'e80a893f'
    ORDER BY "merge_requests"."id" DESC
    LIMIT 1;

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/38507

Migration 20170919211300_remove_temporary_ci_builds_index.rb created a
temporary partial index and tried to drop it at the end of the
migration. In some circumstances apparently it failed to drop the
index and it ended up in our schema.rb.

This accidentally failed to fail due to a bug in the regular
expression for partial indexes which caused the index creation in
schema.rb to be ignored. Now that that's fixed we could be
resurrecting this zombie index from the past in some but not all
databases.

Add a migration to drop this index if it's present to reconcile this
discrepancy.

Missing parens prevented this migration from working in some cases.

Closes #38246

`email_provider` by default is NULL, and if a user had not logged the
value would remain NULL. Upgrading to GitLab 10.0 would lead to a
PG::UniqueViolation because the post-deploy migration would attempt
to reinsert the entry because the NULL comparison between
`users.email_provider` and `user_synced_attributes_metadata.email_provider`
would never match.

Closes #38246

This removes the need for a default scope that adds a "WHERE project_id
= X" clause. This commit also includes an additional index for
Environment#last_deployment, ensuring this query uses just an index scan
instead of also applying a Filter.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/36877

…to stay within our query timeout of 60s. Also reduce the job interval to keep the same overall migration time of ~3.3 days.

This index is required to allow fast retrieval of recent push events
without merge requests. Without this index in place the query would
lead to PostgreSQL scanning over 150 000 index entries all the time,
easily taking up between 200 and 500 milliseconds. This index
reduces the time spent in this process to around 40 milliseconds on
GitLab.com.

This reverts merge request !14240

This file is a duplicate of a post-deploy migration and appears to have
been left in by mistake.

Looping through the Users table in a foreground migration would've been
a bad idea.

[ci skip]

This reverts merge request !14148

This is required as otherwise newly created events will start with the
wrong ID.

Revert "Merge branch '35012-navigation-add-option-to-change-navigation-color-palette' into 'master'"

This reverts merge request !13619

Signed-off-by: NDmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

This finishes the procedure for migrating events from the old format
into the new format. Code no longer uses the old setup and the database
tables used during the migration process are swapped, with the old table
being dropped.

While the database migration can be reversed this will 1) take a lot of
time as data has to be coped around 2) won't restore data in the
"events.data" column as we have no way of restoring this.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/37241