Add latest changes from gitlab-org/gitlab@master

changelogs/unreleased/osw-add-redis-metrics-to-web-requests.yml

+---
+title: Add metrics for Redis usage during web requests
+merge_request: 32605
+author:
+type: added

config/initializers/zz_metrics.rb

@@ -147,6 +147,7 @@ if Gitlab::Metrics.enabled? && !Rails.env.test? && !(Rails.env.development? && d
   Gitlab::Application.configure do |config|
     config.middleware.use(Gitlab::Metrics::RackMiddleware)
     config.middleware.use(Gitlab::Middleware::RailsQueueDuration)
+    config.middleware.use(Gitlab::Metrics::RedisRackMiddleware)
   end
 
   Sidekiq.configure_server do |config|

doc/administration/monitoring/prometheus/gitlab_metrics.md

@@ -92,6 +92,8 @@ The following metrics are available:
 | `gitlab_view_rendering_duration_seconds`                       | Histogram |                   10.2 | Duration for views (histogram)                                                                      | `controller`, `action`, `view`                      |
 | `http_requests_total`                                          | Counter   |                    9.4 | Rack request count                                                                                  | `method`                                            |
 | `http_request_duration_seconds`                                | Histogram |                    9.4 | HTTP response time from rack middleware                                                             | `method`, `status`                                  |
+| `http_redis_requests_duration_seconds`                         | Histogram |                   13.1 | Redis requests duration during web transactions                                                     | `controller`, `action`                              |
+| `http_redis_requests_total`                                    | Counter   |                   13.1 | Redis requests count during web transactions                                                        | `controller`, `action`                              |
 | `pipelines_created_total`                                      | Counter   |                    9.4 | Counter of pipelines created                                                                        |                                                     |
 | `rack_uncaught_errors_total`                                   | Counter   |                    9.4 | Rack connections handling uncaught errors count                                                     |                                                     |
 | `user_session_logins_total`                                    | Counter   |                    9.4 | Counter of how many users have logged in                                                            |                                                     |

doc/ssh/README.md

@@ -119,7 +119,7 @@ Enter file in which to save the key (/home/user/.ssh/id_rsa):
 For guidance, proceed to the [common steps](#common-steps-for-generating-an-ssh-key-pair).
 
 NOTE: **Note:**
-If your have OpenSSH version 7.8 or below, consider the problems associated
+If you have OpenSSH version 7.8 or below, consider the problems associated
 with [encoding](#rsa-keys-and-openssh-from-versions-65-to-78).
 
 ### Common steps for generating an SSH key pair
@@ -370,80 +370,7 @@ git remote set-url origin git@<user_1.gitlab.com>:gitlab-org/gitlab.git
 
 ## Deploy keys
 
-Deploy keys allow read-only or read-write (if enabled) access to one or
-multiple repositories with a single SSH key pair.
-
-This is useful for cloning repositories to your Continuous
-Integration (CI) server. By using deploy keys, you don't have to set up a
-dummy user account.
-
-If you don't have a key pair, you might want to use a
-[deploy token](../user/project/deploy_tokens/index.md#deploy-tokens) instead.
-
-### Per-repository deploy keys
-
-Project maintainers and owners can add a deploy key for a repository.
-
-1. Navigate to the project's **Settings > Repository** page.
-1. Expand the **Deploy Keys** section.
-1. Specify a title for the new deploy key and paste a public SSH key.
-
-After this, the machine that uses the corresponding private SSH key has read-only or
-read-write (if enabled) access to the project.
-
-You can't add the same deploy key twice using the form.
-If you want to add the same key to another project, please enable it in the
-list that says **Deploy keys from projects available to you**. All the deploy
-keys of all the projects you have access to are available. This project
-access can happen through being a direct member of the project, or through
-a group.
-
-Deploy keys can be shared between projects, you just need to add them to each
-project.
-
-### Global shared deploy keys
-
-Global Shared Deploy keys allow read-only or read-write access to
-any repository in the entire GitLab installation.
-
-This is useful for integrating repositories to secured, shared Continuous
-Integration (CI) services or other shared services.
-GitLab administrators can set up the Global Shared Deploy key in GitLab and
-add the private key to any shared systems. Individual repositories opt into
-exposing their repository using these keys when a project maintainers (or higher)
-authorizes a Global Shared Deploy key to be used with their project.
-
-Global Shared Keys can provide greater security compared to Per-Project Deploy
-Keys since an administrator of the target integrated system is the only one
-who needs to know and configure the private key.
-
-GitLab administrators set up Global Deploy keys in the Admin Area under the
-section **Deploy Keys**. Ensure keys have a meaningful title as that will be
-the primary way for project maintainers and owners to identify the correct Global
-Deploy key to add. For instance, if the key gives access to a SaaS CI instance,
-use the name of that service in the key name if that is all it is used for.
-When creating Global Shared Deploy keys, give some thought to the granularity
-of keys - they could be of very narrow usage such as just a specific service or
-of broader usage for something like "Anywhere you need to give read access to
-your repository".
-
-Once a GitLab administrator adds the Global Deployment key, project maintainers
-and owners can add it by:
-
-1. Navigate to the project's **Settings > Repository** page.
-1. Expanding the **Deploy Keys** section.
-1. Clicking **Enable** next to the appropriate key listed under
-   **Public deploy keys available to any project**.
-
-NOTE: **Note:**
-The heading **Public deploy keys available to any project** only appears
-if there is at least one Global Deploy Key configured.
-
-CAUTION: **Warning:**
-Defining Global Deploy Keys does not expose any given repository via
-the key until that repository adds the Global Deploy Key to their project.
-In this way the Global Deploy Keys enable access by other systems, but do
-not implicitly give any access just by setting them up.
+Read the [documentation on Deploy Keys](../user/project/deploy_keys/index.md).
 
 ## Applications
 

doc/user/project/deploy_keys/index.md

+# Deploy Keys
+
+Deploy keys allow read-only or read-write (if enabled) access to one or
+more repositories, by importing an SSH public key to your GitLab instance.
+
+This is useful for cloning repositories to your Continuous
+Integration (CI) server. By using deploy keys, you don't have to set up a
+dummy user account.
+
+There are two types of deploy keys:
+
+- [Project deploy keys](#project-deploy-keys)
+- [Public deploy keys](#public-deploy-keys)
+
+## Key details on deploy keys
+
+Deploy Keys allow a remote machine (VM, physical, and so on) to access a GitLab
+repository with just a few steps. If you want a remote machine to interact with a GitLab
+repository in automation, it's a simple solution.
+
+A drawback is that your repository could become vulnerable if a remote machine is compromised
+by a hacker. You should limit access to the remote machine before a deploy key is
+enabled on your repository. A good rule to follow is to access only to trusted users,
+and make sure that the allowed users have [maintainer permissions or higher](../../permissions.md)
+in the GitLab project.
+
+If this security implication is a concern for your organization,
+[Deploy Tokens](../deploy_tokens/index.md) works as an alternative, but with more
+security control.
+
+## Deploy Keys Permissions
+
+You can choose the access level of a deploy key when you enable it on a project:
+
+- `read-only`: The deploy key can read a repository.
+- `read-write`: The deploy key can read a repository and write to it.
+
+Project maintainers and owners can activate and deactivate deploy keys.
+They can also add their own deploy keys and enable them for this project.
+
+When a `write-access` deploy key is used to push a commit, GitLab checks if
+the **creator** of the deploy key has permission to access the resource. For example:
+
+- When a deploy key is used to push a commit to a [protected branch](../protected_branches.md),
+  the **creator** of the deploy key must have access to the branch.
+- When a deploy key is used to push a commit that triggers a CI/CD pipelines, the **creator** of
+  the deploy key must have access to the CI/CD resources (like protected environments, secret variables, and so on).
+- If the **creator** of a deploy key does not have permissions to read a project's
+  repository, the deploy key _might_ encounter an error during the process.
+
+## Differences between deploy keys and deploy tokens
+
+Both deploy keys and [deploy tokens](../deploy_tokens/index.md#deploy-tokens) can
+help you access a repository, but there are some notables differences between them:
+
+- Deploy keys are shareable between projects that are not related or don't even
+  belong to the same group. Deploy tokens belong to either a project or
+  [a group](../deploy_tokens/index.md#group-deploy-token).
+- A deploy key is an SSH key you need to generate yourself on your machine. A deploy
+  token is generated by your GitLab instance, and is provided to users only once
+  (at creation time).
+- A deploy key is valid as long as it's registered and enabled. Deploy tokens can
+  be time-sensitive, as you can control their validity by setting an expiration date to them.
+- You can't log in to a registry with deploy keys, or perform read / write operations
+  on it, but this [is possible with deploy tokens](../deploy_tokens/index.md#gitlab-deploy-token).
+- You need an SSH key pair to use deploy keys, but not deploy tokens.
+
+## How to enable Deploy Keys
+
+### Project deploy keys
+
+[Project maintainers and owners](../../permissions.md#project-members-permissions)
+can add or enable a deploy key for a project repository:
+
+1. Navigate to the project's **Settings > Repository** page.
+1. Expand the **Deploy Keys** section.
+1. Specify a title for the new deploy key and paste your public SSH key.
+1. (Optional) Check **Write access allowed** to allow `read-write` access. Leave it unchecked for `read-only` access.
+
+There are three lists of Project Deploy Keys:
+
+- Enabled deploy keys
+- Privately accessible deploy keys
+- Public accessible deploy keys
+
+![Deploy Keys section](img/deploy_keys_v13_0.png)
+
+After you add a key, it will be enabled for this project by default, and it'll appear
+in the **Enabled deploy keys** tab.
+
+In the **Privately accessible deploy keys** tab, you can enable a private key which
+has been already imported in a different project. If you have access to these keys,
+it's because you have either:
+
+- Previously uploaded the keys yourself in a different project.
+- You are a maintainer or owner of the other project where the keys were imported.
+
+In the **Publicly accessible deploy keys** tab, you can enable
+keys that were [made available to your entire GitLab instance](#public-deploy-keys).
+
+After a key is added, you can edit it to update its title, or switch between `read-only`
+and `read-write` access.
+
+NOTE: **Note:**
+If you have enabled a privately or publicly accessible or deploy key for your
+project, and if you then update the access level for this key from `read-only` to
+`read-write`, the change will be only for the **current project**.
+
+### Public deploy keys
+
+Public deploy keys allow `read-only` or `read-write`
+access to any repository in your GitLab instance. This is useful for integrating
+repositories to secure, shared services, such as CI/CD.
+
+Instance administrators can add public deploy keys:
+
+1. Go to **Admin Area** (**{admin}**) **> Deploy Keys**.
+1. Click on **New deploy key**.
+
+   Make sure your new key has a meaningful title, as it is the primary way for project
+   maintainers and owners to identify the correct public deploy key to add. For example,
+   if the key gives access to a SaaS CI/CD instance, use the name of that service
+   in the key name if that is all the key is used for.
+
+![Public Deploy Keys section](img/public_deploy_key_v13_0.png)
+
+After adding a key, it will be available to any shared systems. Project maintainers
+or higher can [authorize a public deploy key](#project-deploy-keys) to start using it with the project.
+
+NOTE: **Note:**
+The **Publicly accessible deploy keys** tab within Project's CI/CD settings only appears
+if there is at least one Public deploy key configured.
+
+Public deploy keys can provide greater security compared to project deploy keys, as
+the administrator of the target integrated system is the only one who needs to know the key value,
+or configure it.
+
+When creating a Public deploy key, determine whether or not it can be defined for
+very narrow usage, such as just a specific service, or if it needs to be defined for
+broader usage, such as full `read-write` access for all services.
+
+CAUTION: **Warning:**
+Adding a public deploy key does not immediately expose any repository to it. Public
+deploy keys enable access from other systems, but access is not given to any project
+until a project maintainer chooses to make use of it.
+
+## Troubleshooting
+
+### Deploy Key cannot push to a protected branch
+
+If the owner of this deploy key does not have access to a [protected
+branch](../protected_branches.md), then this deploy key won't have access to
+the branch either. In addition to this, choosing the **No one** value in
+[the "Allowed to push" section](../protected_branches.md#configuring-protected-branches)
+means that no users **and** no services using deploy keys can push to that selected branch.
+
+Refer to [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30769) for more information.

lib/backup/files.rb

@@ -78,7 +78,8 @@ module Backup
       return if status.compact.all?(&:success?)
 
       regex = /^g?tar: \.: Cannot mkdir: No such file or directory$/
-      raise Backup::Error, 'Backup failed' unless err_r.read =~ regex
+      error = err_r.read
+      raise Backup::Error, "Backup failed. #{error}" unless error =~ regex
     end
   end
 end

lib/gitlab/metrics/redis_rack_middleware.rb

+# frozen_string_literal: true
+
+module Gitlab
+  module Metrics
+    # Rack middleware for tracking Redis metrics from Grape and Web requests.
+    class RedisRackMiddleware
+      def initialize(app)
+        @app = app
+      end
+
+      def call(env)
+        transaction = Gitlab::Metrics.current_transaction
+
+        @app.call(env)
+      ensure
+        record_metrics(transaction) if transaction
+      end
+
+      private
+
+      def record_metrics(transaction)
+        labels = transaction.labels
+        query_time = Gitlab::Instrumentation::Redis.query_time
+        request_count = Gitlab::Instrumentation::Redis.get_request_count
+
+        Gitlab::Metrics.counter(:http_redis_requests_total,
+                                'Amount of calls to Redis servers during web requests',
+                                Gitlab::Metrics::Transaction::BASE_LABELS).increment(labels, request_count)
+
+        Gitlab::Metrics.histogram(:http_redis_requests_duration_seconds,
+                                  'Query time for Redis servers during web requests',
+                                  Gitlab::Metrics::Transaction::BASE_LABELS,
+                                  Gitlab::Instrumentation::Redis::QUERY_TIME_BUCKETS).observe(labels, query_time)
+      end
+    end
+  end
+end

spec/lib/gitlab/metrics/redis_rack_middleware_spec.rb

+# frozen_string_literal: true
+
+require 'spec_helper'
+
+describe Gitlab::Metrics::RedisRackMiddleware do
+  let(:app) { double(:app) }
+  let(:middleware) { described_class.new(app) }
+  let(:env) { {} }
+  let(:transaction) { Gitlab::Metrics::WebTransaction.new(env) }
+
+  before do
+    allow(app).to receive(:call).with(env).and_return('wub wub')
+  end
+
+  describe '#call' do
+    context 'when metrics are disabled' do
+      before do
+        allow(Gitlab::Metrics).to receive(:current_transaction).and_return(nil)
+      end
+
+      it 'calls the app' do
+        expect(middleware.call(env)).to eq('wub wub')
+      end
+
+      it 'does not record metrics' do
+        expect(Gitlab::Metrics).not_to receive(:counter)
+        expect(Gitlab::Metrics).not_to receive(:histogram)
+
+        middleware.call(env)
+      end
+    end
+
+    context 'when metrics are enabled' do
+      let(:counter) { double(Prometheus::Client::Counter, increment: nil) }
+      let(:histogram) { double(Prometheus::Client::Histogram, observe: nil) }
+      let(:redis_query_time) { 0.1 }
+      let(:redis_requests_count) { 2 }
+
+      before do
+        allow(Gitlab::Instrumentation::Redis).to receive(:query_time) { redis_query_time }
+        allow(Gitlab::Instrumentation::Redis).to receive(:get_request_count) { redis_requests_count }
+
+        allow(Gitlab::Metrics).to receive(:counter)
+          .with(:http_redis_requests_total,
+                an_instance_of(String),
+                Gitlab::Metrics::Transaction::BASE_LABELS)
+          .and_return(counter)
+
+        allow(Gitlab::Metrics).to receive(:histogram)
+          .with(:http_redis_requests_duration_seconds,
+                an_instance_of(String),
+                Gitlab::Metrics::Transaction::BASE_LABELS,
+                Gitlab::Instrumentation::Redis::QUERY_TIME_BUCKETS)
+          .and_return(histogram)
+
+        allow(Gitlab::Metrics).to receive(:current_transaction).and_return(transaction)
+      end
+
+      it 'calls the app' do
+        expect(middleware.call(env)).to eq('wub wub')
+      end
+
+      it 'records redis metrics' do
+        expect(counter).to receive(:increment).with(transaction.labels, redis_requests_count)
+        expect(histogram).to receive(:observe).with(transaction.labels, redis_query_time)
+
+        middleware.call(env)
+      end
+
+      it 'records redis metrics if an error is raised' do
+        expect(counter).to receive(:increment).with(transaction.labels, redis_requests_count)
+        expect(histogram).to receive(:observe).with(transaction.labels, redis_query_time)
+
+        allow(app).to receive(:call).with(env).and_raise(StandardError)
+
+        expect { middleware.call(env) }.to raise_error(StandardError)
+      end
+    end
+  end
+end

spec/support/shared_examples/models/concerns/blob_replicator_strategy_shared_examples.rb

-# frozen_string_literal: true
-
-# Include these shared examples in specs of Replicators that include
-# BlobReplicatorStrategy.
-#
-# A let variable called model_record should be defined in the spec. It should be
-# a valid, unpersisted instance of the model class.
-#
-RSpec.shared_examples 'a blob replicator' do
-  include EE::GeoHelpers
-
-  let_it_be(:primary) { create(:geo_node, :primary) }
-  let_it_be(:secondary) { create(:geo_node) }
-
-  subject(:replicator) { model_record.replicator }
-
-  before do
-    stub_current_geo_node(primary)
-  end
-
-  describe '#handle_after_create_commit' do
-    it 'creates a Geo::Event' do
-      expect do
-        replicator.handle_after_create_commit
-      end.to change { ::Geo::Event.count }.by(1)
-
-      expect(::Geo::Event.last.attributes).to include(
-        "replicable_name" => replicator.replicable_name, "event_name" => "created", "payload" => { "model_record_id" => replicator.model_record.id })
-    end
-
-    it 'schedules the checksum calculation if needed' do
-      expect(Geo::BlobVerificationPrimaryWorker).to receive(:perform_async)
-      expect(replicator).to receive(:needs_checksum?).and_return(true)
-
-      replicator.handle_after_create_commit
-    end
-
-    it 'does not schedule the checksum calculation if feature flag is disabled' do
-      stub_feature_flags(geo_self_service_framework: false)
-
-      expect(Geo::BlobVerificationPrimaryWorker).not_to receive(:perform_async)
-      allow(replicator).to receive(:needs_checksum?).and_return(true)
-
-      replicator.handle_after_create_commit
-    end
-  end
-
-  describe '#calculate_checksum!' do
-    it 'calculates the checksum' do
-      model_record.save!
-
-      replicator.calculate_checksum!
-
-      expect(model_record.reload.verification_checksum).not_to be_nil
-      expect(model_record.reload.verified_at).not_to be_nil
-    end
-
-    it 'saves the error message and increments retry counter' do
-      model_record.save!
-
-      allow(model_record).to receive(:calculate_checksum!) do
-        raise StandardError.new('Failure to calculate checksum')
-      end
-
-      replicator.calculate_checksum!
-
-      expect(model_record.reload.verification_failure).to eq 'Failure to calculate checksum'
-      expect(model_record.verification_retry_count).to be 1
-    end
-  end
-
-  describe '#consume_created_event' do
-    it 'invokes Geo::BlobDownloadService' do
-      service = double(:service)
-
-      expect(service).to receive(:execute)
-      expect(::Geo::BlobDownloadService).to receive(:new).with(replicator: replicator).and_return(service)
-
-      replicator.consume_event_created
-    end
-  end
-
-  describe '#carrierwave_uploader' do
-    it 'is implemented' do
-      expect do
-        replicator.carrierwave_uploader
-      end.not_to raise_error
-    end
-  end
-
-  describe '#model' do
-    let(:invoke_model) { replicator.class.model }
-
-    it 'is implemented' do
-      expect do
-        invoke_model
-      end.not_to raise_error
-    end
-
-    it 'is a Class' do
-      expect(invoke_model).to be_a(Class)
-    end
-
-    # For convenience (and reliability), instead of asking developers to include shared examples on each model spec as well
-    context 'replicable model' do
-      it 'defines #replicator' do
-        expect(model_record).to respond_to(:replicator)
-      end
-
-      it 'invokes replicator.handle_after_create_commit on create' do
-        expect(replicator).to receive(:handle_after_create_commit)
-
-        model_record.save!
-      end
-    end
-  end
-end