# Geo self-service framework (alpha)

> 原文：[https://docs.gitlab.com/ee/development/geo/framework.html](https://docs.gitlab.com/ee/development/geo/framework.html)

*   [Nomenclature](#nomenclature)
*   [Geo Domain-Specific Language](#geo-domain-specific-language)
    *   [The replicator](#the-replicator)
    *   [Linking to a model](#linking-to-a-model)
    *   [API](#api)
        *   [Library](#library)
*   [Existing Replicator Strategies](#existing-replicator-strategies)
    *   [Blob Replicator Strategy](#blob-replicator-strategy)
        *   [Replication](#replication)
        *   [Verification](#verification)
        *   [Metrics](#metrics)
        *   [GraphQL API](#graphql-api)
        *   [Admin UI](#admin-ui)

# Geo self-service framework (alpha)[](#geo-self-service-framework-alpha "Permalink")

**注意：**本文档可能会随时更改. 这是我们正在研究的建议，一旦实施完成，此文档将得到更新. 跟随[史诗般的](https://gitlab.com/groups/gitlab-org/-/epics/2161)进度.**注意：** Geo 自助服务框架当前处于 Alpha 状态. 如果您需要复制新的数据类型，请与 Geo 小组联系以讨论选项. 您可以在 Slack 的`#g_geo`与他们联系，或在问题或合并请求中提及`@geo-team` .

Geo 提供了一个 API，使跨 Geo 节点轻松复制数据类型成为可能. 该 API 以 Ruby 域特定语言（DSL）的形式呈现，旨在使创建数据类型的工程师只需花费很少的精力即可复制数据.

## Nomenclature[](#nomenclature "Permalink")

在深入研究 API 之前，开发人员需要了解一些特定于地理位置的命名约定.

Model

模型是活动模型，在整个 Rails 代码库中都是如此. 它通常与数据库表绑定. 从地理角度来看，模型可以具有一个或多个资源.

Resource

资源是属于模型的一条数据，由 GitLab 功能生成. 使用存储机制将其持久化. 默认情况下，资源不可复制.

Data type

Data type is how a resource is stored. Each resource should fit in one of the data types Geo supports: :- Git repository :- Blob :- Database

有关更多详细信息，请参见[数据类型](../../administration/geo/replication/datatypes.html) .

Geo Replicable

可复制资源是 Geo 希望在 Geo 节点之间同步的资源. 受支持的可复制数据类型有限. 实现属于已知数据类型之一的资源的复制所需的工作量很小.

Geo Replicator

地理复制器是知道如何复制可复制对象的对象. 它负责：：-触发事件（生产者）：-消费事件（消费者）

它与 Geo Replicable 数据类型相关. 所有复制器都有一个公共接口，可用于处理（即产生和使用）事件. 它负责主节点（产生事件的地方）和次节点（消耗事件的地方）之间的通信. 想要将 Geo 纳入其功能的工程师将使用复制器的 API 来实现这一目标.

Geo Domain-Specific Language

语法糖使工程师可以轻松指定应复制哪些资源以及如何复制.

## Geo Domain-Specific Language[](#geo-domain-specific-language "Permalink")

### The replicator[](#the-replicator "Permalink")

首先，您需要编写一个复制器. 复制器位于[`ee/app/replicators/geo`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/app/replicators/geo) . 对于每个需要复制的资源，即使多个资源绑定到同一模型，也应指定一个单独的复制器.

例如，以下复制器复制软件包文件：

```
module Geo
  class PackageFileReplicator < Gitlab::Geo::Replicator
    # Include one of the strategies your resource needs
    include ::Geo::BlobReplicatorStrategy

    # Specify the CarrierWave uploader needed by the used strategy
    def carrierwave_uploader
      model_record.file
    end

    # Specify the model this replicator belongs to
    def self.model
      ::Packages::PackageFile
    end
  end
end 
```

类名应该是唯一的. 它还与注册表的表名紧密相关，因此在此示例中，注册表表将为`package_file_registry` .

对于不同的数据类型，Geo 支持包括不同的策略. 选择一个适合您的需求.

### Linking to a model[](#linking-to-a-model "Permalink")

要将此复制器绑定到模型，需要在模型代码中添加以下内容：

```
class Packages::PackageFile < ApplicationRecord
  include ::Gitlab::Geo::ReplicableModel

  with_replicator Geo::PackageFileReplicator
end 
```

### API[](#api "Permalink")

设置好后，可以通过模型轻松访问复制器：

```
package_file = Packages::PackageFile.find(4) # just a random id as example
replicator = package_file.replicator 
```

或者从复制器取回模型：

```
replicator.model_record
=> <Packages::PackageFile id:4> 
```

复制器可用于生成事件，例如在 ActiveRecord 挂钩中：

```
 after_create_commit -> { replicator.publish_created_event } 
```

#### Library[](#library "Permalink")

所有这些背后的框架位于[`ee/lib/gitlab/geo/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/gitlab/geo) .

## Existing Replicator Strategies[](#existing-replicator-strategies "Permalink")

在编写一种新的复制器策略之前，请检查以下内容，以查看现有策略之一是否已经可以处理您的资源. 如果不确定，请咨询地理团队.

### Blob Replicator Strategy[](#blob-replicator-strategy "Permalink")

使用`Geo::BlobReplicatorStrategy`模块，Geo 可以轻松支持使用[CarrierWave 的](https://github.com/carrierwaveuploader/carrierwave) `Uploader::Base`模型.

首先，每个文件应具有其自己的主要 ID 和模型. Geo 强烈建议将*每个文件都*视为头等公民，因为根据我们的经验，这大大简化了跟踪复制和验证状态.

例如，要添加对具有`Widget` `widgets`表的`Widget`模型引用的文件的支持，您将执行以下步骤：

#### Replication[](#replication "Permalink")

1.  在`Widget`类中包含`Gitlab::Geo::ReplicableModel` ，并使用`with_replicator Geo::WidgetReplicator`指定 Replicator 类.

    此时， `Widget`类应如下所示：

    ```
    # frozen_string_literal: true

    class Widget < ApplicationRecord
      include ::Gitlab::Geo::ReplicableModel

      with_replicator Geo::WidgetReplicator

      mount_uploader :file, WidgetUploader

      def self.replicables_for_geo_node
        # Should be implemented. The idea of the method is to restrict
        # the set of synced items depending on synchronization settings
      end
      ...
    end 
    ```

2.  创建`ee/app/replicators/geo/widget_replicator.rb` . 实现`#carrierwave_uploader`方法，该方法应返回`CarrierWave::Uploader` . 并实现类方法`.model`以返回`Widget`类.

    ```
    # frozen_string_literal: true

    module Geo
      class WidgetReplicator < Gitlab::Geo::Replicator
        include ::Geo::BlobReplicatorStrategy

        def self.model
          ::Widget
        end

        def carrierwave_uploader
          model_record.file
        end
      end
    end 
    ```

3.  创建`ee/spec/replicators/geo/widget_replicator_spec.rb`并执行必要的设置，以定义共享示例的`model_record`变量.

    ```
    # frozen_string_literal: true

    require 'spec_helper'

    RSpec.describe Geo::WidgetReplicator do
      let(:model_record) { build(:widget) }

      it_behaves_like 'a blob replicator'
    end 
    ```

4.  创建`widget_registry`表，以便 Geo 次要对象可以跟踪每个 Widget 文件的同步和验证状态：

    ```
    # frozen_string_literal: true

    class CreateWidgetRegistry < ActiveRecord::Migration[6.0]
      DOWNTIME = false

      disable_ddl_transaction!

      def up
        unless table_exists?(:widget_registry)
          ActiveRecord::Base.transaction do
            create_table :widget_registry, id: :bigserial, force: :cascade do |t|
              t.integer :widget_id, null: false
              t.integer :state, default: 0, null: false, limit: 2
              t.integer :retry_count, default: 0, limit: 2
              t.text :last_sync_failure
              t.datetime_with_timezone :retry_at
              t.datetime_with_timezone :last_synced_at
              t.datetime_with_timezone :created_at, null: false

              t.index :widget_id
              t.index :retry_at
              t.index :state
            end
          end
        end

        add_text_limit :widget_registry, :last_sync_failure, 255
      end

      def down
        drop_table :widget_registry
      end
    end 
    ```

5.  Create `ee/app/models/geo/widget_registry.rb`:

    ```
    # frozen_string_literal: true

    class Geo::WidgetRegistry < Geo::BaseRegistry
      include Geo::ReplicableRegistry

      MODEL_CLASS = ::Widget
      MODEL_FOREIGN_KEY = :widget_id

      belongs_to :widget, class_name: 'Widget'
    end 
    ```

    方法`has_create_events?` 在大多数情况下应该返回`true` . 但是，如果您添加的实体没有创建事件，则根本不要添加该方法.

6.  Update `REGISTRY_CLASSES` in `ee/app/workers/geo/secondary/registry_consistency_worker.rb`.

7.  Create `ee/spec/factories/geo/widget_registry.rb`:

    ```
    # frozen_string_literal: true

    FactoryBot.define do
      factory :geo_widget_registry, class: 'Geo::WidgetRegistry' do
        widget
        state { Geo::WidgetRegistry.state_value(:pending) }

        trait :synced do
          state { Geo::WidgetRegistry.state_value(:synced) }
          last_synced_at { 5.days.ago }
        end

        trait :failed do
          state { Geo::WidgetRegistry.state_value(:failed) }
          last_synced_at { 1.day.ago }
          retry_count { 2 }
          last_sync_failure { 'Random error' }
        end

        trait :started do
          state { Geo::WidgetRegistry.state_value(:started) }
          last_synced_at { 1.day.ago }
          retry_count { 0 }
        end
      end
    end 
    ```

8.  Create `ee/spec/models/geo/widget_registry_spec.rb`:

    ```
    # frozen_string_literal: true

    require 'spec_helper'

    RSpec.describe Geo::WidgetRegistry, :geo, type: :model do
      let_it_be(:registry) { create(:geo_widget_registry) }

      specify 'factory is valid' do
        expect(registry).to be_valid
      end

      include_examples 'a Geo framework registry'

      describe '.find_registry_differences' do
        ... # To be implemented
      end
    end 
    ```

小部件现在应该由 Geo 复制！

#### Verification[](#verification "Permalink")

1.  将验证状态字段添加到`widgets`表中，以便 Geo 主数据库可以跟踪验证状态：

    ```
    # frozen_string_literal: true

    class AddVerificationStateToWidgets < ActiveRecord::Migration[6.0]
      DOWNTIME = false

      def change
        add_column :widgets, :verification_retry_at, :datetime_with_timezone
        add_column :widgets, :verified_at, :datetime_with_timezone
        add_column :widgets, :verification_checksum, :binary, using: 'verification_checksum::bytea'
        add_column :widgets, :verification_failure, :string
        add_column :widgets, :verification_retry_count, :integer
      end
    end 
    ```

2.  在`verification_failure`和`verification_checksum`上添加部分索引，以确保可以高效执行重新验证：

    ```
    # frozen_string_literal: true

    class AddVerificationFailureIndexToWidgets < ActiveRecord::Migration[6.0]
      include Gitlab::Database::MigrationHelpers

      DOWNTIME = false

      disable_ddl_transaction!

      def up
        add_concurrent_index :widgets, :verification_failure, where: "(verification_failure IS NOT NULL)", name: "widgets_verification_failure_partial"
        add_concurrent_index :widgets, :verification_checksum, where: "(verification_checksum IS NOT NULL)", name: "widgets_verification_checksum_partial"
      end

      def down
        remove_concurrent_index :widgets, :verification_failure
        remove_concurrent_index :widgets, :verification_checksum
      end
    end 
    ```

要做的事情：在二级服务器上添加验证. 这应作为以下内容的一部分完成[：Geo：自助服务框架-包文件验证的首次实现](https://gitlab.com/groups/gitlab-org/-/epics/1817)

小部件现在应由 Geo 验证！

#### Metrics[](#metrics "Permalink")

指标由`Geo::MetricsUpdateWorker`收集，保存在`GeoNodeStatus`以显示在 UI 中，然后发送给 Prometheus.

1.  将字段`widget_count` ， `widget_checksummed_count` ， `widget_checksum_failed_count` ， `widget_synced_count` ， `widget_failed_count`和`widget_registry_count`到`ee/app/models/geo_node_status.rb` `GeoNodeStatus#RESOURCE_STATUS_FIELDS`数组中.
2.  将相同的字段添加到`ee/app/models/geo_node_status.rb` `GeoNodeStatus#PROMETHEUS_METRICS`哈希中.
3.  将相同字段添加到`doc/administration/monitoring/prometheus/gitlab_metrics.md` `Sidekiq metrics`表中.
4.  将相同的字段添加到`doc/api/geo_nodes.md` `GET /geo_nodes/status`示例响应中.
5.  将相同的字段添加到`ee/spec/models/geo_node_status_spec.rb`和`ee/spec/factories/geo_node_statuses.rb` .
6.  Set `widget_count` in `GeoNodeStatus#load_data_from_current_node`:

    ```
    self.widget_count = Geo::WidgetReplicator.primary_total_count 
    ```

7.  添加`GeoNodeStatus#load_widgets_data`来设置`widget_synced_count` ， `widget_failed_count`和`widget_registry_count` ：

    ```
    def load_widget_data
      self.widget_synced_count = Geo::WidgetReplicator.synced_count
      self.widget_failed_count = Geo::WidgetReplicator.failed_count
      self.widget_registry_count = Geo::WidgetReplicator.registry_count
    end 
    ```

8.  Call `GeoNodeStatus#load_widgets_data` in `GeoNodeStatus#load_secondary_data`.

9.  Set `widget_checksummed_count` and `widget_checksum_failed_count` in `GeoNodeStatus#load_verification_data`:

    ```
    self.widget_checksummed_count = Geo::WidgetReplicator.checksummed_count   self.widget_checksum_failed_count = Geo::WidgetReplicator.checksum_failed_count 
    ```

小部件复制和验证指标现在应该可以在 API，管理区域 UI 和 Prometheus 中使用！

#### GraphQL API[](#graphql-api "Permalink")

1.  在`ee/app/graphql/types/geo/geo_node_type.rb`向`GeoNodeType`添加一个新字段：

    ```
    field :widget_registries, ::Types::Geo::WidgetRegistryType.connection_type,
          null: true,
          resolver: ::Resolvers::Geo::WidgetRegistriesResolver,
          description: 'Find widget registries on this Geo node',
          feature_flag: :geo_self_service_framework 
    ```

2.  新添加`widget_registries`字段名的`expected_fields`在阵列`ee/spec/graphql/types/geo/geo_node_type_spec.rb` .

3.  Create `ee/app/graphql/resolvers/geo/widget_registries_resolver.rb`:

    ```
    # frozen_string_literal: true

    module Resolvers
      module Geo
        class WidgetRegistriesResolver < BaseResolver
          include RegistriesResolver
        end
      end
    end 
    ```

4.  Create `ee/spec/graphql/resolvers/geo/widget_registries_resolver_spec.rb`:

    ```
    # frozen_string_literal: true

    require 'spec_helper'

    RSpec.describe Resolvers::Geo::WidgetRegistriesResolver do
      it_behaves_like 'a Geo registries resolver', :geo_widget_registry
    end 
    ```

5.  Create `ee/app/finders/geo/widget_registry_finder.rb`:

    ```
    # frozen_string_literal: true

    module Geo
      class WidgetRegistryFinder
        include FrameworkRegistryFinder
      end
    end 
    ```

6.  Create `ee/spec/finders/geo/widget_registry_finder_spec.rb`:

    ```
    # frozen_string_literal: true

    require 'spec_helper'

    RSpec.describe Geo::WidgetRegistryFinder do
      it_behaves_like 'a framework registry finder', :geo_widget_registry
    end 
    ```

7.  Create `ee/app/graphql/types/geo/widget_registry_type.rb`:

    ```
    # frozen_string_literal: true

    module Types
      module Geo
        # rubocop:disable Graphql/AuthorizeTypes because it is included
        class WidgetRegistryType < BaseObject
          include ::Types::Geo::RegistryType

          graphql_name 'WidgetRegistry'
          description 'Represents the sync and verification state of a widget'

          field :widget_id, GraphQL::ID_TYPE, null: false, description: 'ID of the Widget'
        end
      end
    end 
    ```

8.  Create `ee/spec/graphql/types/geo/widget_registry_type_spec.rb`:

    ```
    # frozen_string_literal: true

    require 'spec_helper'

    RSpec.describe GitlabSchema.types['WidgetRegistry'] do
      it_behaves_like 'a Geo registry type'

      it 'has the expected fields (other than those included in RegistryType)' do
        expected_fields = %i[widget_id]

        expect(described_class).to have_graphql_fields(*expected_fields).at_least
      end
    end 
    ```

9.  Add integration tests for providing Widget registry data to the frontend via the GraphQL API, by duplicating and modifying the following shared examples in `ee/spec/requests/api/graphql/geo/registries_spec.rb`:

    ```
    it_behaves_like 'gets registries for', {
      field_name: 'widgetRegistries',
      registry_class_name: 'WidgetRegistry',
      registry_factory: :geo_widget_registry,
      registry_foreign_key_field_name: 'widgetId'
    } 
    ```

现在应该可以通过 GraphQL API 获得各个小部件同步和验证数据！

1.  注意复制"更新"事件. Geo Framework 目前不支持复制"更新"事件，因为此时添加到框架的所有实体都是不可变的. 如果您要添加的实体属于这种情况，请遵循[https://gitlab.com/gitlab-org/gitlab/-/issues/118743](https://gitlab.com/gitlab-org/gitlab/-/issues/118743)和[https://gitlab.com/gitlab-org/gitlab /// issues / 118745](https://gitlab.com/gitlab-org/gitlab/-/issues/118745)作为添加新事件类型的示例. 添加通知后，请同时删除它.

#### Admin UI[](#admin-ui "Permalink")

要做的事情：这应该作为《 [地理手册》的](https://gitlab.com/groups/gitlab-org/-/epics/2525)一部分完成[：实现自助服务框架可复制的前端](https://gitlab.com/groups/gitlab-org/-/epics/2525)

窗口小部件同步和验证数据（总计和个人）现在应该在管理界面中可用！