These are regular Rails migrations that are executed by default. A user
can opt-out of these migrations by setting an environment variable
during the deployment process.

Fixes gitlab-org/gitlab-ce#22133

Code improvements, bug fixes, finish documentation and specs

mail_room was configured to deliver mail to the `incoming_email`
queue while `EmailReceiveWorker` was reading the `email_receiver`
queue. Adds a migration that repeats the work of a previous
migration to ensure all mails that wound up in the old
queue get processed.

Closes #23689

Dumping too many jobs in the same queue (e.g. the "default" queue) is a
dangerous setup. Jobs that take a long time to process can effectively
block any other work from being performed given there are enough of
these jobs.

Furthermore it becomes harder to monitor the jobs as a single queue
could contain jobs for different workers. In such a setup the only
reliable way of getting counts per job is to iterate over all jobs in a
queue, which is a rather time consuming process.

By using separate queues for various workers we have better control over
throughput, we can add weight to queues, and we can monitor queues
better. Some workers still use the same queue whenever their work is
related. For example, the various CI pipeline workers use the same
"pipeline" queue.

This commit includes a Rails migration that moves Sidekiq jobs from the
old queues to the new ones. This migration also takes care of doing the
inverse if ever needed. This does require downtime as otherwise new jobs
could be scheduled in the old queues after this migration completes.

This commit also includes an RSpec test that blacklists the use of the
"default" queue and ensures cron workers use the "cronjob" queue.

Fixes gitlab-org/gitlab-ce#23370

The previous implementation of the migration failed on staging because
the migration was attempted to remove labels from projects that did not
actually have duplicates. This occurred because the SQL query did not
account for the project ID when selecting the labels.

To replicate the problem:

1. Disable the uniqueness validation in app/models/label.rb.
2. Create a duplicate label "bug" in project A.
3. Create the same label in project B with label "bug".

The migration will attempt to remove the label in B even if there are no duplicates.

Closes #23609

Signed-off-by: NRémy Coutable <remy@rymai.me>

This commit introduces a Sidekiq worker that precalculates the list of
trending projects on a daily basis. The resulting set is stored in a
database table that is then queried by Project.trending.

This setup means that Unicorn workers no longer _may_ have to calculate
the list of trending projects. Furthermore it supports filtering without
any complex caching mechanisms.

The data in the "trending_projects" table is inserted in the same order
as the project ranking. This means that getting the projects in the
correct order is simply a matter of:

    SELECT projects.*
    FROM projects
    INNER JOIN trending_projects ON trending_projects.project_id = projects.id
    ORDER BY trending_projects.id ASC;

Such a query will only take a few milliseconds at most (as measured on
GitLab.com), opposed to a few seconds for the query used for calculating
the project ranks.

The migration in this commit does not require downtime and takes care of
populating an initial list of trending projects.

This commit adds a number of _html columns and, with the exception of Note,
starts updating them whenever the content of their partner fields changes.

Note has a collision with the note_html attr_accessor; that will be fixed later

A background worker for clearing these cache columns is also introduced - use
`rake cache:clear` to set it off. You can clear the database or Redis caches
separately by running `rake cache:clear:db` or `rake cache:clear:redis`,
respectively.

Changes include:

- Ensure Member.add_user is not called directly when not necessary
- New GroupMember.add_users_to_group to have the same abstraction level as for Project
- Refactor Member.add_user to take a source instead of an array of members
- Fix Rubocop offenses
- Always use Project#add_user instead of project.team.add_user
- Factorize users addition as members in Member.add_users_to_source
- Make access_level a keyword argument in GroupMember.add_users_to_group and ProjectMember.add_users_to_projects
- Destroy any requester before adding them as a member
- Improve the way we handle access requesters in Member.add_user
  Instead of removing the requester and creating a new member,
  we now simply accepts their access request. This way, they will
  receive a "access request granted" email.
- Fix error that was previously silently ignored
- Stop raising when access level is invalid in Member, let Rails validation do their work
Signed-off-by: NRémy Coutable <remy@rymai.me>

* Add newline to user organization spec according to test guide
* Remove unnecessary comments from user organization database migration
Signed-off-by: NDmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

Signed-off-by: NDmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

It seems this wasn't updated properly when migrations were added in a
previous commit.

- Don't use `TableReferences` - using `.arel_table` is shorter!
- Move some database-related code to `Gitlab::Database`
- Remove the `MergeRequest#issues_closed` and
  `Issue#closed_by_merge_requests`  associations. They were either
  shadowing or were too similar to existing methods. They are not being
  used anywhere, so it's better to remove them to reduce confusion.
- Use Rails 3-style validations
- Index for `MergeRequest::Metrics#first_deployed_to_production_at`
- Only include `CycleAnalyticsHelpers::TestGeneration` for specs that
  need it.
- Other minor refactorings.

Incorrect syntax.

1. Change multiple updates to a single `update_all`

2. Use cascading deletes

3. Extract an average function for the database median.

4. Move database median to `lib/gitlab/database`

5. Use `delete_all` instead of `destroy_all`

6. Minor refactoring

1. Add indexes to `CreateMergeRequestsClosingIssues` columns.
2. Remove an extraneous `check_if_open` check that is redundant now.

It would've been better to rebase this in, but that's not possible
because more people are working on this branch.