Add latest changes from gitlab-org/gitlab@master

doc/administration/server_hooks.md

@@ -11,12 +11,13 @@ disqus_identifier: 'https://docs.gitlab.com/ee/administration/custom_hooks.html'
 > - Server hooks must be configured on the filesystem of the GitLab server. Only GitLab server administrators will be able to complete these tasks. Please explore [webhooks](../user/project/integrations/webhooks.md) and [GitLab CI/CD](../ci/README.md) as an option if you do not have filesystem access. For a user-configurable Git hook interface, see [Push Rules](../push_rules/push_rules.md), available in GitLab Starter **(STARTER)**.
 > - Server hooks won't be replicated to secondary nodes if you use [GitLab Geo](geo/replication/index.md).
 
-Git natively supports hooks that are executed on different actions.
-Examples of server-side Git hooks include pre-receive, post-receive, and update.
-See [Git SCM Server-Side Hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks#Server-Side-Hooks) for more information about each hook type.
+Git natively supports hooks that are executed on different actions. These hooks run
+on the server and can be used to enforce specific commit policies or perform other
+tasks based on the state of the repository.
 
-As of GitLab Shell version 2.2.0 (which requires GitLab 7.5+), GitLab
-administrators can add custom Git hooks to any GitLab project.
+Examples of server-side Git hooks include `pre-receive`, `post-receive`, and `update`.
+See [Git SCM Server-Side Hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks#Server-Side-Hooks)
+for more information about each hook type.
 
 ## Create a server hook for a repository
 

doc/ci/pipelines/index.md

@@ -367,10 +367,14 @@ evaluates the job names: `\d+[\s:\/\\]+\d+\s*`.
 When running manual jobs you can supply additional job specific variables.
 
 You can do this from the job page of the manual job you want to run with
-additional variables.
-
-This is useful when you want to alter the execution of a job by using
-environment variables.
+additional variables. To access this page, click on the **name** of the manual job in
+the pipeline view, *not* the play (**{play}**) button.
+
+This is useful when you want to alter the execution of a job that uses
+[custom environment variables](../variables/README.md#custom-environment-variables).
+Adding a variable name (key) and value here will override the value defined in
+[the UI or `.gitlab-ci.yml`](../variables/README.md#creating-a-custom-environment-variable),
+for a single run of the manual job.
 
 ![Manual job variables](img/manual_job_variables.png)
 

doc/development/pipelines.md

@@ -489,6 +489,71 @@ for more information.
 
 Consult the [Review Apps](testing_guide/review_apps.md) dedicated page for more information.
 
+## Pre-clone step
+
+The `gitlab-org/gitlab` project on GitLab.com uses a [pre-clone step](https://gitlab.com/gitlab-org/gitlab/issues/39134)
+to seed the project with a recent archive of the repository. This is done for
+several reasons:
+
+- It speeds up builds because a 800 MB download only takes seconds, as opposed to a full Git clone.
+- It significantly reduces load on the file server, as smaller deltas mean less time spent in `git pack-objects`.
+
+The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable
+[defined by GitLab.com shared runners](../user/gitlab_com/index.md#pre-clone-script).
+
+The `CI_PRE_CLONE_SCRIPT` is currently defined as a project CI/CD
+variable:
+
+```shell
+echo "Downloading archived master..."
+wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master.tar.gz
+
+if [ ! -f /tmp/gitlab.tar.gz ]; then
+    echo "Repository cache not available, cloning a new directory..."
+    exit
+fi
+
+rm -rf $CI_PROJECT_DIR
+echo "Extracting tarball into $CI_PROJECT_DIR..."
+mkdir -p $CI_PROJECT_DIR
+cd $CI_PROJECT_DIR
+tar xzf /tmp/gitlab.tar.gz
+rm -f /tmp/gitlab.tar.gz
+chmod a+w $CI_PROJECT_DIR
+```
+
+The first step of the script downloads `gitlab-master.tar.gz` from
+Google Cloud Storage. There is a [GitLab CI job named `cache-repo`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/cache-repo.gitlab-ci.yml#L5)
+that is responsible for keeping that archive up-to-date. Every two hours
+on a scheduled pipeline, it does the following:
+
+1. Creates a fresh clone of the `gitlab-org/gitlab` repository on GitLab.com.
+1. Saves the data as a `.tar.gz`.
+1. Uploads it into the Google Cloud Storage bucket.
+
+When a CI job runs with this configuration, you'll see something like
+this:
+
+```shell
+$ eval "$CI_PRE_CLONE_SCRIPT"
+Downloading archived master...
+Extracting tarball into /builds/group/project...
+Fetching changes...
+Reinitialized existing Git repository in /builds/group/project/.git/
+```
+
+Note that the `Reinitialized existing Git repository` message shows that
+the pre-clone step worked. The runner runs `git init`, which
+overwrites the Git configuration with the appropriate settings to fetch
+from the GitLab repository.
+
+`CI_REPO_CACHE_CREDENTIALS` contains the Google Cloud service account
+JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. These
+credentials are stored in the 1Password GitLab.com Production vault.
+
+Note that this bucket should be located in the same continent as the
+runner, or [network egress charges will apply](https://cloud.google.com/storage/pricing).
+
 ---
 
 [Return to Development documentation](README.md)

doc/user/gitlab_com/index.md

@@ -144,6 +144,26 @@ Below are the shared Runners settings.
 | Default Docker image                  | `ruby:2.5`                                        | -          |
 | `privileged` (run [Docker in Docker](https://hub.docker.com/_/docker/)) | `true`          | `false`    |
 
+#### Pre-clone script
+
+Linux Shared Runners on GitLab.com provide a way to run commands in a CI
+job before the Runner attempts to run `git init` and `git fetch` to
+download a GitLab repository. The
+[pre_clone_script](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section)
+can be used for:
+
+- Seeding the build directory with repository data
+- Sending a request to a server
+- Downloading assets from a CDN
+- Any other commands that must run before the `git init`
+
+To use this feature, define a [CI/CD variable](../../ci/variables/README.md#via-the-ui) called
+`CI_PRE_CLONE_SCRIPT` that contains a bash script.
+
+[This example](../../development/pipelines.md#pre-clone-step)
+demonstrates how you might use a pre-clone step to seed the build
+directory.
+
 #### `config.toml`
 
 The full contents of our `config.toml` are:
@@ -164,6 +184,7 @@ sentry_dsn = "X"
   request_concurrency = X
   url = "https://gitlab.com/"
   token = "SHARED_RUNNER_TOKEN"
+  pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\""
   executor = "docker+machine"
   environment = [
     "DOCKER_DRIVER=overlay2",