When a project is forked, the new repository used to be a deep copy of everything
stored on disk by leveraging `git clone`. This works well, and makes isolation
between repository easy. However, the clone is at the start 100% the same as the
origin repository. And in the case of the objects in the object directory, this
is almost always going to be a lot of duplication.

Object Pools are a way to create a third repository that essentially only exists
for its 'objects' subdirectory. This third repository's object directory will be
set as alternate location for objects. This means that in the case an object is
missing in the local repository, git will look in another location. This other
location is the object pool repository.

When Git performs garbage collection, it's smart enough to check the
alternate location. When objects are duplicated, it will allow git to
throw one copy away. This copy is on the local repository, where to pool
remains as is.

These pools have an origin location, which for now will always be a
repository that itself is not a fork. When the root of a fork network is
forked by a user, the fork still clones the full repository. Async, the
pool repository will be created.

Either one of these processes can be done earlier than the other. To
handle this race condition, the Join ObjectPool operation is
idempotent. Given its idempotent, we can schedule it twice, with the
same effect.

To accommodate the holding of state two migrations have been added.
1. Added a state column to the pool_repositories column. This column is
managed by the state machine, allowing for hooks on transitions.
2. pool_repositories now has a source_project_id. This column in
convenient to have for multiple reasons: it has a unique index allowing
the database to handle race conditions when creating a new record. Also,
it's nice to know who the host is. As that's a short link to the fork
networks root.

Object pools are only available for public project, which use hashed
storage and when forking from the root of the fork network. (That is,
the project being forked from itself isn't a fork)

In this commit message I use both ObjectPool and Pool repositories,
which are alike, but different from each other. ObjectPool refers to
whatever is on the disk stored and managed by Gitaly. PoolRepository is
the record in the database.

This removes the `ForkedProjectLink` model that has been replaced by
the `ForkNetworkMember` and `ForkNetwork` combination. All existing
relations have been adjusted to use these new models.

The `forked_project_link` table has been dropped.

The "Forks" count on the admin dashboard has been updated to count all
`ForkNetworkMember` rows and deduct the number of `ForkNetwork`
rows. This is because now the "root-project" of a fork network also
has a `ForkNetworkMember` row. This count could become inaccurate when
the root of a fork network is deleted.

A forked project stores its LFS objects in the `forked_from_project`.
So the LFS objects become inaccessible, and therefore delete them from
the database so they'll get cleaned up.

To be refactored when implementing
https://gitlab.com/gitlab-org/gitlab-ce/issues/39769

Direct disk access is done through Gitaly now, so the legacy path was
deprecated. This path was used in Gitlab::Shell however. This required
the refactoring in this commit.

Added is the removal of direct path access on the project model, as that
lookup wasn't needed anymore is most cases.

Closes https://gitlab.com/gitlab-org/gitaly/issues/1111

Prior to this change, this method was called add_namespace, which broke
the CRUD convention and made it harder to grep for what I was looking
for. Given the change was a find and replace kind of fix, this was
changed without opening an issue and on another feature branch.

If more dynamic calls are made to add_namespace, these could've been
missed which might lead to incorrect bahaviour. However, going through
the commit log it seems thats not the case.

In this case the project will get the minimum between both
visibilities.

If that visibility is restricted, then a lower level will be picked.

The helper creates a fork of a project with all provided attributes,
but skipping the creation of the repository on disk.

When no fork network exists for the source projects, we create a new
one with the correct source

There are some redundancies in the validation steps, and that is to
preserve current error messages behavior

Also few specs have to be changed in order to fix madness in validation
logic.

The number of forks of a project doesn't change very frequently and
running a COUNT(*) every time this information is requested can be quite
expensive. We also end up running such a COUNT(*) query at least twice
on the homepage of a project.

By caching this data and refreshing it when necessary we can reduce
project homepage loading times by around 60 milliseconds (based on the
timings of https://gitlab.com/gitlab-org/gitlab-ce).

Remove superfluous lib: true, type: redis, service: true, models: true, services: true, no_db: true, api: true
Signed-off-by: NRémy Coutable <remy@rymai.me>

Projects::ForkService delegates to this service almost entirely, but needed
one small change so it would propagate create errors correctly.

CreateService#execute needs significant refactoring; it is now right at the
complexity limit set by Rubocop. I avoided doing so in this commit to keep the
diff as small as possible.

Several tests depend on the insecure behaviour of ForkService, so fi them up at
the same time.

This reverts commit 3e991230.

# Conflicts:
#	app/models/project.rb

- Enable CI by default for all new projects

Signed-off-by: NDmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

Signed-off-by: NDmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

Use import_status to track async import status and give feedback to the user

Closes #2388
Closes #2400

Add gitlab-shell to error message to give user a clue that something may be wrong there.

Ran into this in #2082. User was told that repositories were created when they were
not due to hooks symlink being wrong.

Signed-off-by: NJeroen van Baarsen <jeroenvanbaarsen@gmail.com>

Remove overload of BaseService.initialize, so initialize gains params,
which is used to pass the namespace (like e.g. in TransferService).

The namespace is checked for permission to create projects in it.

Signed-off-by: NDmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

Signed-off-by: NDmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>