From 6adcd827a75c33efb299199ed94acccbfade67d9 Mon Sep 17 00:00:00 2001 From: GitLab Bot Date: Thu, 2 Jul 2020 03:08:59 +0000 Subject: [PATCH] Add latest changes from gitlab-org/gitlab@master --- doc/administration/gitaly/praefect.md | 2 - doc/ci/yaml/README.md | 46 ++++++++++++++++--- doc/raketasks/cleanup.md | 4 +- .../reducing_the_repo_size_using_git.md | 27 ++++++++++- .../requests/snippet_shared_examples.rb | 7 --- 5 files changed, 67 insertions(+), 19 deletions(-) diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 4bc36393cf2..13ec67068dd 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -131,8 +131,6 @@ with secure tokens as you complete the setup process. Praefect cluster directly; that could lead to data loss. 1. `PRAEFECT_SQL_PASSWORD`: this password is used by Praefect to connect to PostgreSQL. -1. `GRAFANA_PASSWORD`: this password is used to access the `admin` - account in the Grafana dashboards. We will note in the instructions below where these secrets are required. diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 89a7278aaa4..c09157fd7ff 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -383,7 +383,7 @@ NOTE: **Note:** The configuration is a snapshot in time and persisted in the database. Any changes to referenced `.gitlab-ci.yml` configuration won't be reflected in GitLab until the next pipeline is created. -The files defined in `include` are: +The files defined by `include` are: - Deep merged with those in `.gitlab-ci.yml`. - Always evaluated first and merged with the content of `.gitlab-ci.yml`, @@ -391,11 +391,11 @@ The files defined in `include` are: TIP: **Tip:** Use merging to customize and override included CI/CD configurations with local -definitions. +definitions. Local definitions in `.gitlab-ci.yml` will override included definitions. NOTE: **Note:** -Using YAML aliases across different YAML files sourced by `include` is not -supported. You must only refer to aliases in the same file. Instead +Using [YAML anchors](#anchors) across different YAML files sourced by `include` is not +supported. You must only refer to anchors in the same file. Instead of using YAML anchors, you can use the [`extends` keyword](#extends). #### `include:local` @@ -972,24 +972,38 @@ spinach: ``` In GitLab 12.0 and later, it's also possible to use multiple parents for -`extends`. The algorithm used for merge is "closest scope wins", so -keys from the last member will always shadow anything defined on other +`extends`. + +#### Merge details + +`extends` is able to merge hashes but not arrays. +The algorithm used for merge is "closest scope wins", so +keys from the last member will always override anything defined on other levels. For example: ```yaml .only-important: + variables: + URL: "http://my-url.internal" + IMPORTANT_VAR: "the details" only: - master - stable tags: - production + script: + - echo "Hello world!" .in-docker: + variables: + URL: "http://docker-url.internal" tags: - docker image: alpine rspec: + variables: + GITLAB: "is-awesome" extends: - .only-important - .in-docker @@ -1001,6 +1015,10 @@ This results in the following `rspec` job: ```yaml rspec: + variables: + URL: "http://docker-url.internal" + IMPORTANT_VAR: "the details" + GITLAB: "is-awesome" only: - master - stable @@ -1011,6 +1029,15 @@ rspec: - rake rspec ``` +Note that in the example above: + +- `variables` sections have been merged but that `URL: "http://my-url.internal"` +has been overwritten by `URL: "http://docker-url.internal"`. +- `tags: ['production']` has been overwritten by `tags: ['docker']`. +- `script` has not been merged but rather `script: ['echo "Hello world!"']` has + been overwritten by `script: ['rake rspec']`. Arrays can be + merged using [YAML anchors](#anchors). + #### Using `extends` and `include` together `extends` works across configuration files combined with `include`. @@ -4149,6 +4176,10 @@ of `.gitlab-ci.yml`. Read more about the various [YAML features](https://learnxinyminutes.com/docs/yaml/). +In most cases, the [`extends` keyword](#extends) is more user friendly and should +be used over these special YAML features. YAML anchors may still +need to be used to merge arrays. + ### Anchors > Introduced in GitLab 8.6 and GitLab Runner v1.1.1. @@ -4273,7 +4304,8 @@ You can see that the hidden jobs are conveniently used as templates. NOTE: **Note:** You can't use YAML anchors across multiple files when leveraging the [`include`](#include) -feature. Anchors are only valid within the file they were defined in. +feature. Anchors are only valid within the file they were defined in. Instead +of using YAML anchors, you can use the [`extends` keyword](#extends). #### YAML anchors for `before_script` and `after_script` diff --git a/doc/raketasks/cleanup.md b/doc/raketasks/cleanup.md index 76c51bab6f8..0e64119da3a 100644 --- a/doc/raketasks/cleanup.md +++ b/doc/raketasks/cleanup.md @@ -2,7 +2,7 @@ GitLab provides Rake tasks for cleaning up GitLab instances. -## Remove unreferenced LFS files from filesystem +## Remove unreferenced LFS files > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36628) in GitLab 12.10. @@ -42,7 +42,7 @@ Note that this Rake task only removes the references to LFS files. Unreferenced later (once a day). If you need to garbage collect them immediately, run `rake gitlab:cleanup:orphan_lfs_files` described below. -## Remove unreferenced LFS files +### Remove unreferenced LFS files immediately > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36628) in GitLab 12.10. diff --git a/doc/user/project/repository/reducing_the_repo_size_using_git.md b/doc/user/project/repository/reducing_the_repo_size_using_git.md index 6f4e83544c7..b70b4c3c70d 100644 --- a/doc/user/project/repository/reducing_the_repo_size_using_git.md +++ b/doc/user/project/repository/reducing_the_repo_size_using_git.md @@ -25,6 +25,11 @@ Rewriting repository history is a destructive operation. Make sure to backup you you begin. The best way back up a repository is to [export the project](../settings/import_export.md#exporting-a-project-and-its-data). +NOTE: **Note:** +Git LFS files can only be removed by an Administrator using a +[Rake task](../../../raketasks/cleanup.md). Removal of this limitation +[is planned](https://gitlab.com/gitlab-org/gitlab/-/issues/223621). + ## Purge files from repository history To make cloning your project faster, rewrite branches and tags to remove unwanted files. @@ -40,12 +45,25 @@ To make cloning your project faster, rewrite branches and tags to remove unwante 1. Using `git filter-repo`, purge any files from the history of your repository. - To purge all large files, the `--strip-blobs-bigger-than` option can be used: + To purge large files, the `--strip-blobs-bigger-than` option can be used: ```shell git filter-repo --strip-blobs-bigger-than 10M ``` + To purge large files stored using Git LFS, the `--blob--callback` option can + be used. The example below, uses the callback to read the file size from the + Git LFS pointer, and removes files larger than 10MB. + + ```shell + git filter-repo --blob-callback ' + if blob.data.startswith(b"version https://git-lfs.github.com/spec/v1"): + size_in_bytes = int.from_bytes(blob.data[124:], byteorder="big") + if size_in_bytes > 10*1000: + blob.skip() + ' + ``` + To purge specific large files by path, the `--path` and `--invert-paths` options can be combined: ```shell @@ -80,6 +98,12 @@ To make cloning your project faster, rewrite branches and tags to remove unwante [Protected tags](../protected_tags.md) will cause this to fail. To proceed, you must remove tag protection, push, and then re-enable protected tags. +1. Manually run [project housekeeping](../../../administration/housekeeping.md#manual-housekeeping) + +NOTE: **Note** +Project statistics are cached for performance. You may need to wait 5-10 minutes +to see a reduction in storage utilization. + ## Purge files from GitLab storage To reduce the size of your repository in GitLab, you must remove GitLab internal references to @@ -176,6 +200,7 @@ You will receive an email once it has completed. When using repository cleanup, note: +- Project statistics are cached. You may need to wait 5-10 minutes to see a reduction in storage utilization. - Housekeeping prunes loose objects older than 2 weeks. This means objects added in the last 2 weeks will not be removed immediately. If you have access to the [Gitaly](../../../administration/gitaly/index.md) server, you may run `git gc --prune=now` to diff --git a/spec/support/shared_examples/requests/snippet_shared_examples.rb b/spec/support/shared_examples/requests/snippet_shared_examples.rb index dc1988a9b46..67303f8ca4c 100644 --- a/spec/support/shared_examples/requests/snippet_shared_examples.rb +++ b/spec/support/shared_examples/requests/snippet_shared_examples.rb @@ -74,13 +74,6 @@ RSpec.shared_examples 'update with repository actions' do end end -RSpec.shared_examples 'snippet response without repository URLs' do - it 'skip inclusion of repository URLs' do - expect(json_response).not_to have_key('ssh_url_to_repo') - expect(json_response).not_to have_key('http_url_to_repo') - end -end - RSpec.shared_examples 'snippet blob content' do it 'returns content from repository' do subject -- GitLab