# 为现有集群增加白屏监控

OBD 自 V1.6.0 开始支持 prometheus 和 grafana 组件的部署，若您想为之前部署的集群新增白屏监控，可参考本文进行操作。

本文分为三个场景进行介绍，您可根据根据自身集群情况选择合适的场景参考。

<main id="notice" type='explain'>
  <h4>说明</h4>
  <p>本文所提供的配置示例仅供参考，详细配置及配置说明请切换到 <code>/usr/obd/example</code> 目录下查看对应组件的示例。</p>
</main>

## 场景一：现有集群未部署 OBAgent

当您现有集群未部署 OBAgent 时，若要增加白屏监控，需新建一个包含 obagent、prometheus 和 grafana 组件的集群。

其中，OBAgent 通过单独的配置指定用于采集 OceanBase 数据库的监控信息；在配置文件中声明 Prometheus 依赖于 obagent 组件，Grafana 依赖于 prometheus 组件。

具体配置文件示例如下：

```yaml
# user:
#   username: your username
#   password: your password if need
#   key_file: your ssh-key file path if need
#   port: your ssh port, default 22
#   timeout: ssh connection timeout (second), default 30
obagent:
  servers:
    # Please don't use hostname, only IP can be supported
    - 192.168.1.2
    - 192.168.1.3
    - 192.168.1.4
  global:
    # The working directory for obagent. obagent is started under this directory. This is a required field.
    home_path: /root/obagent
    # The port that pulls and manages the metrics. The default port number is 8088.
    server_port: 8088
    # Debug port for pprof. The default port number is 8089.
    pprof_port: 8089
    # Log level. The default value is INFO.
    log_level: INFO
    # Log path. The default value is log/monagent.log.
    log_path: log/monagent.log
    # Encryption method. OBD supports aes and plain. The default value is plain.
    crypto_method: plain
    # Path to store the crypto key. The default value is conf/.config_secret.key.
    # crypto_path: conf/.config_secret.key
    # Size for a single log file. Log size is measured in Megabytes. The default value is 30M.
    log_size: 30
    # Expiration time for logs. The default value is 7 days.
    log_expire_day: 7
    # The maximum number for log files. The default value is 10.
    log_file_count: 10
    # Whether to use local time for log files. The default value is true.
    # log_use_localtime: true
    # Whether to enable log compression. The default value is true.
    # log_compress: true
    # Username for HTTP authentication. The default value is admin.
    http_basic_auth_user: ******
    # Password for HTTP authentication. The default value is root.
    http_basic_auth_password: ******
    # Username for debug service. The default value is admin.
    pprof_basic_auth_user: ******
    # Password for debug service. The default value is root.
    pprof_basic_auth_password: ******
    # Monitor username for OceanBase Database. The user must have read access to OceanBase Database as a system tenant. The default value is root.
    monitor_user: root
    # Monitor password for OceanBase Database. The default value is empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the root_password in oceanbase-ce.
    monitor_password: 
    # The SQL port for observer. The default value is 2881. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the mysql_port in oceanbase-ce.
    sql_port: 2881
    # The RPC port for observer. The default value is 2882. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the rpc_port in oceanbase-ce.
    rpc_port: 2882
    # Cluster name for OceanBase Database. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the appname in oceanbase-ce.
    cluster_name: obcluster
    # Cluster ID for OceanBase Database. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the cluster_id in oceanbase-ce.
    cluster_id: 1
    # Monitor status for OceanBase Database.  Active is to enable. Inactive is to disable. The default value is active. When you deploy an cluster automatically, OBD decides whether to enable this parameter based on depends.
    ob_monitor_status: active
    # Monitor status for your host. Active is to enable. Inactive is to disable. The default value is active.
    host_monitor_status: active
    # Whether to disable the basic authentication for HTTP service. True is to disable. False is to enable. The default value is false.
    disable_http_basic_auth: false
    # Whether to disable the basic authentication for the debug interface. True is to disable. False is to enable. The default value is false.
    disable_pprof_basic_auth: false
    # Synchronize the obagent-related information to the specified path of the remote host, as the targets specified by `file_sd_config` in the Prometheus configuration.
    # For prometheus that depends on obagent, it can be specified to $home_path/targets of prometheus.
    # For independently deployed prometheus, specify the files to be collected by setting `config` -> `scrape_configs` -> `file_sd_configs` -> `files`.For details, please refer to prometheus-only-example.yaml.
    # target_sync_configs:
    #   - host: 192.168.1.1
    #     target_dir: /root/prometheus/targets
    #     username: your username
    #     password: your password if need
    #     key_file: your ssh-key file path if need
    #     port: your ssh port, default 22
    #     timeout: ssh connection timeout (second), default 30
  192.168.1.2:
    # Zone name for your observer. The default value is zone1. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the zone name in oceanbase-ce.
    zone_name: zone1
  192.168.1.3:
    # Zone name for your observer. The default value is zone1. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the zone name in oceanbase-ce.
    zone_name: zone2
  192.168.1.4:
    # Zone name for your observer. The default value is zone1. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the zone name in oceanbase-ce.
    zone_name: zone3
prometheus:
  depends:
    - obagent
  servers:
    - 192.168.1.5
  global:
    home_path: /root/prometheus
grafana:
  depends:
    - prometheus
  servers:
    - 192.168.1.5
  global:
    home_path: /root/grafana
    login_password: *********
```

修改配置文件后，执行如下命令部署并启动新集群：

```shell
obd cluster deploy <new deploy name> -c new_config.yaml
obd cluster start <new deploy name>
```

集群启动后，根据展示信息访问 Grafana 页面，即可查看现有集群的监控信息。

## 场景二：现有集群已部署 OBAgent

当您现有集群已部署 OBAgent 时，若要增加白屏监控，需新建一个包含 prometheus 和 grafana 组件的集群。

此时，由于无法声明 depends obagent，所以需要手动进行关联。查看原有集群中 OBAgent 安装目录下 `conf/prometheus_config/prometheus.yaml` 文件，将对应的配置复制到新集群配置中 prometheus 组件的 `global` -> `conifg` 下。具体配置示例如下：

```yaml
# user:
#   username: your username
#   password: your password if need
#   key_file: your ssh-key file path if need
#   port: your ssh port, default 22
#   timeout: ssh connection timeout (second), default 30
prometheus:
  servers:
    - 192.168.1.5
  global:
    # The working directory for prometheus. prometheus is started under this directory. This is a required field.
    home_path: /root/prometheus
    config: # Configuration of the Prometheus service. The format is consistent with the Prometheus config file. Corresponds to the `config.file` parameter.
      global:
        scrape_interval: 1s
        evaluation_interval: 10s

      rule_files:
        - "rules/*rules.yaml"

      scrape_configs:
        - job_name: prometheus
          metrics_path: /metrics
          scheme: http
          static_configs:
            - targets:
                - 'localhost:9090'
        - job_name: node
          basic_auth:
            username: ******
            password: ******
          metrics_path: /metrics/node/host
          scheme: http
          static_configs:
            - targets:
                - 192.168.1.2:8088
        - job_name: ob_basic
          basic_auth:
            username: ******
            password: ******
          metrics_path: /metrics/ob/basic
          scheme: http
          static_configs:
            - targets:
                - 192.168.1.2:8088
        - job_name: ob_extra
          basic_auth:
            username: ******
            password: ******
          metrics_path: /metrics/ob/extra
          scheme: http
          static_configs:
            - targets:
                - 192.168.1.2:8088
        - job_name: agent
          basic_auth:
            username: ******
            password: ******
          metrics_path: /metrics/stat
          scheme: http
          static_configs:
            - targets:
                - 192.168.1.2:8088
grafana:
  servers:
    - 192.168.1.5
  depends:
    - prometheus
  global:
    home_path: /root/grafana
    login_password: ********* # Grafana login password. The default value is 'oceanbase'.
```

<main id="notice" type='explain'>
  <h4>说明</h4>
  <p>上文配置文件示例中 <code>basic_auth</code> 配置项的用户名和密码需和 OBAgent 配置文件中 http_basic_auth_xxx 配置项相对应。</p>
</main>

修改配置文件后，执行如下命令部署新集群：

```shell
obd cluster deploy <new deploy name> -c new_config.yaml
```

部署完成后，将 OBAgent 安装目录下 `conf/prometheus_config/rules` 目录复制到 Prometheus 的安装目录下。

执行如下命令启动新集群：

```shell
obd cluster start <new deploy name>
```

集群启动后，根据展示信息访问 Grafana 页面，即可查看现有集群的监控信息。

<main id="notice" type='notice'>
  <h4>注意</h4>
  <ol>
  <li>
  <p><code>scrape_configs</code> 中的 Prometheus 监控项中 <code>'localhost:9090'</code> 需要根据当前 Prometheus 的监听地址进行修改，如果当前 Prometheus 开启了认证，也需要对应的配置 <code>basic_auth</code>。这里提到的监听地址为部署 Prometheus 的地址，即 Prometheus 配置中的 address 和 port 配置项。</p>
  </li>
  <li>
  <p>如果原有集群的 obagent 节点有所变化，需通过 <code>obd cluster edit-config <new deploy name></code> 同步 OBAgent 安装目录下 <code>conf/prometheus_config/prometheus.yaml</code> 的内容。</p>
  </li>
  </ol>
</main>

## 场景三：监控多个集群并动态同步 OBAgent 变化

如若希望 Prometheus 可以采集多个集群的监控信息，或者动态同步 OBAgent 的变化，可在场景二的基础上做少许改动。

将 Prometheus 配置中的 `static_configs` 替换为 `file_sd_config` 来获取和同步 OBAgent 的节点信息。如下示例中表示收集 Prometheus 安装目录（`home_path`）下 targets 目录下的所有 yaml 文件。

<main id="notice" type='explain'>
  <h4>说明</h4>
  <p>Prometheus 安装目录下的 targets 目录需在原有集群配置文件中 obagent 组件下进行相关配置才会被创建。具体配置参见 <a href="#修改被监控集群配置">修改被监控集群配置</a>。</p>
</main>

```yaml
# user:
#   username: your username
#   password: your password if need
#   key_file: your ssh-key file path if need
#   port: your ssh port, default 22
#   timeout: ssh connection timeout (second), default 30
prometheus:
  servers:
    - 192.168.1.5
  global:
    # The working directory for prometheus. prometheus is started under this directory. This is a required field.
    home_path: /root/prometheus
    config: # Configuration of the Prometheus service. The format is consistent with the Prometheus config file. Corresponds to the `config.file` parameter.
      global:
        scrape_interval: 1s
        evaluation_interval: 10s

      rule_files:
        - "rules/*rules.yaml"

      scrape_configs:
        - job_name: prometheus
          metrics_path: /metrics
          scheme: http
          static_configs:
            - targets:
                - 'localhost:9090'
        - job_name: node
          basic_auth:
            username: ******
            password: ******
          metrics_path: /metrics/node/host
          scheme: http
          file_sd_configs:
            - files:
              - targets/*.yaml
        - job_name: ob_basic
          basic_auth:
            username: ******
            password: ******
          metrics_path: /metrics/ob/basic
          scheme: http
          file_sd_configs:
            - files:
              - targets/*.yaml
        - job_name: ob_extra
          basic_auth:
            username: ******
            password: ******
          metrics_path: /metrics/ob/extra
          scheme: http
          file_sd_configs:
            - files:
              - targets/*.yaml
        - job_name: agent
          basic_auth:
            username: ******
            password: ******
          metrics_path: /metrics/stat
          scheme: http
          file_sd_configs:
            - files:
              - targets/*.yaml
grafana:
  servers:
    - 192.168.1.5
  depends:
    - prometheus
  global:
    home_path: /root/grafana
    login_password: ********* # Grafana login password. The default value is 'oceanbase'.
```

<main id="notice" type='explain'>
  <h4>说明</h4>
  <p>上文配置文件示例中 <code>basic_auth</code> 配置项的用户名和密码需和 OBAgent 配置文件中 http_basic_auth_xxx 配置项相对应。</p>
</main>

修改配置文件后，执行如下命令部署新集群：

```shell
obd cluster deploy <new deploy name> -c new_config.yaml
```

部署完成后，将 OBAgent 安装目录下的 `conf/prometheus_config/rules` 目录复制到 Prometheus 的安装目录下。

执行如下命令启动新集群：

```shell
obd cluster start <new deploy name>
```

部署好新集群后，根据展示信息访问 Grafana 页面，此时无法查看被监控集群的监控信息，还需修改被监控集群的 obagent 配置。

### 修改被监控集群配置

若要在 Prometheus 安装目录下创建 targets 目录，需执行 `obd cluster edit-config <deploy name>` 命令修改配置文件，在配置文件中增加配置项 `target_sync_configs` 指向 Prometheus 安装目录下的 targets 目录（默认会使用当前集群的 user 配置，如果有差别可以按照示例进行设置）。

```yaml
obagent:
  servers:
    # Please don't use hostname, only IP can be supported
    - 192.168.1.2
    - 192.168.1.3
    - 192.168.1.4
  global:
    ....
    target_sync_configs:
      - host: 192.168.1.5
        target_dir: /root/prometheus/targets
    #    username: your username
    #    password: your password if need
    #    key_file: your ssh-key file path if need
    #    port: your ssh port, default 22
    #    timeout: ssh connection timeout (second), default 30
    ...
```

修改配置文件后需根据提示重启集群，重启后访问 Grafana 页面，即可查看现有集群的监控信息。

<main id="notice" type='notice'>
  <h4>注意</h4>
  <ol>
  <li>
  <p><code>scrape_configs</code> 中的 Prometheus 监控项中 <code>'localhost:9090'</code> 需要根据当前 Prometheus 的监听地址进行修改，如果当前 Prometheus 开启了认证，也需要对应的配置 <code>basic_auth</code>。这里提到的监听地址为部署 Prometheus 的地址，即 Prometheus 配置中的 address 和 port 配置项。</p>
  </li>
  <li>
  <p>被 Prometheus 采集的 OBAgent 的 http 用户名密码需要保持一致，如果不一致，请拆分采集项。</p>
  </li>
  </ol>
</main>
