未验证 提交 d3206b1b 编写于 作者: kimmking's avatar kimmking 提交者: GitHub

Merge pull request #5601 from menghaoranss/doc5.x-metadata

Add metadata center documents for 5.x
......@@ -7,4 +7,4 @@ chapter = true
## 导览
编排治理模块提供配置中心/注册中心(以及规划中的元数据中心)、配置动态化、数据库熔断禁用、调用链路等治理能力。
编排治理模块提供配置中心/注册中心/元数据中心、配置动态化、数据库熔断禁用、调用链路等治理能力。
......@@ -7,4 +7,4 @@ chapter = true
## Navigation
Provide orchestration of config/registry center(and metadata center in plan), dynamic configuration, database & application disabling, and APM.
Provide orchestration of config/registry/metadata center, dynamic configuration, database & application disabling, and APM.
+++
pre = "<b>3.3.4. </b>"
pre = "<b>3.3.5. </b>"
title = "应用性能监控集成"
weight = 4
weight = 5
+++
## 背景
......
+++
pre = "<b>3.3.4. </b>"
pre = "<b>3.3.5. </b>"
title = "APM Integration"
weight = 4
weight = 5
+++
## Background
......
......@@ -48,7 +48,7 @@ sql.show: true
多个数据库连接池的集合,不同数据库连接池属性自适配(例如:DBCP,C3P0,Druid, HikariCP)。
```yaml
ds_0: !!org.apache.shardingsphere.orchestration.yaml.YamlDataSourceConfiguration
ds_0: !!org.apache.shardingsphere.orchestration.core.configuration.YamlDataSourceConfiguration
dataSourceClassName: com.zaxxer.hikari.HikariDataSource
properties:
url: jdbc:mysql://127.0.0.1:3306/demo_ds_0?serverTimezone=UTC&useSSL=false
......@@ -60,7 +60,7 @@ ds_0: !!org.apache.shardingsphere.orchestration.yaml.YamlDataSourceConfiguration
minPoolSize: 1
username: root
maxLifetimeMilliseconds: 1800000
ds_1: !!org.apache.shardingsphere.orchestration.yaml.YamlDataSourceConfiguration
ds_1: !!org.apache.shardingsphere.orchestration.core.configuration.YamlDataSourceConfiguration
dataSourceClassName: com.zaxxer.hikari.HikariDataSource
properties:
url: jdbc:mysql://127.0.0.1:3306/demo_ds_1?serverTimezone=UTC&useSSL=false
......
......@@ -48,7 +48,7 @@ sql.show: true
A collection of multiple database connection pools, whose properties (e.g. DBCP, C3P0, Druid and HikariCP) are configured by users themselves.
```yaml
ds_0: !!org.apache.shardingsphere.orchestration.yaml.YamlDataSourceConfiguration
ds_0: !!org.apache.shardingsphere.orchestration.core.configuration.YamlDataSourceConfiguration
dataSourceClassName: com.zaxxer.hikari.HikariDataSource
properties:
url: jdbc:mysql://127.0.0.1:3306/demo_ds_0?serverTimezone=UTC&useSSL=false
......@@ -60,7 +60,7 @@ ds_0: !!org.apache.shardingsphere.orchestration.yaml.YamlDataSourceConfiguration
minPoolSize: 1
username: root
maxLifetimeMilliseconds: 1800000
ds_1: !!org.apache.shardingsphere.orchestration.yaml.YamlDataSourceConfiguration
ds_1: !!org.apache.shardingsphere.orchestration.core.configuration.YamlDataSourceConfiguration
dataSourceClassName: com.zaxxer.hikari.HikariDataSource
properties:
url: jdbc:mysql://127.0.0.1:3306/demo_ds_1?serverTimezone=UTC&useSSL=false
......
+++
pre = "<b>3.3.5. </b>"
pre = "<b>3.3.6. </b>"
title = "数据脱敏"
weight = 5
weight = 6
+++
## 背景
......
+++
pre = "<b>3.3.5. </b>"
pre = "<b>3.3.6. </b>"
title = "Data Masking"
weight = 5
weight = 6
+++
## Background
......
+++
pre = "<b>3.3.3. </b>"
title = "元数据中心"
weight = 3
+++
## 实现动机
- 元数据是Sharding-JDBC/Sharding-Proxy所使用的数据源的表、列和索引等核心数据,元数据保障ShardingSphere各个组件能够正确运行。
- 元数据中心对元数据进行统一组织和管理,实现元数据的统一加载、变更通知和数据同步。
## 元数据中心数据结构
元数据中心在定义的命名空间和治理节点的metadata节点下,以YAML格式存储,每个逻辑数据源独立存储。
```
├─orchestration-namespace
│ ├─orchestration-name
│ │ ├──metadata
│ │ │ ├──schema1
│ │ │ │ ├── [YAML text contents]
│ │ │ ├──schema2
│ │ │ │ ├── [YAML text contents]
│ │ │ ├──....
```
## YAML Text Contents
在元数据中心中,元数据分为 `configuredSchemaMetaData``unconfiguredSchemaMetaDataMap` 两部分。
元数据内容目前不支持动态修改。
```
configuredSchemaMetaData:
tables: # 表
t_order: # 表名
columns: # 列
id: # 列名
caseSensitive: false
dataType: 0
generated: false
name: id
primaryKey: trues
order_id:
caseSensitive: false
dataType: 0
generated: false
name: order_id
primaryKey: false
indexs: # 索引
t_user_order_id_index: # 索引名
name: t_user_order_id_index
t_order_item:
columns:
order_id:
caseSensitive: false
dataType: 0
generated: false
name: order_id
primaryKey: false
unconfiguredSchemaMetaDataMap:
ds_0: # 数据源
tables: # 表
t_user: # 表名
columns: # 列
user_id: # 列名
caseSensitive: false
dataType: 0
generated: false
name: user_id
primaryKey: false
id:
caseSensitive: false
dataType: 0
generated: false
name: id
primaryKey: true
order_id:
caseSensitive: false
dataType: 0
generated: false
name: order_id
primaryKey: false
indexes: # 索引
t_user_order_id_index: # 索引名
name: t_user_order_id_index
t_user_user_id_index:
name: t_user_user_id_index
primary:
name: PRIMARY
```
### configuredSchemaMetaData
存储所有配置了分片规则的数据源的元数据。
### unconfiguredSchemaMetaDataMap
存储没有配置分片规则的数据源的元数据。
## 变更通知
通过某一个Proxy实例执行DDL以后,ShardingSphere先将新的元数据存储到元数据中心,然后通过事件广播机制,通知其它Proxy实例从元数据中心同步元数据,保证元数据一致。
+++
pre = "<b>3.3.3. </b>"
title = "Metadata Center"
weight = 3
+++
## Motivation
- Metadata is the core data of the data source used by Sharding-JDBC/Sharding-Proxy, which contains tables, columns, and indexes, etc. Metadata ensures that each component of ShardingSphere can run correctly.
- The metadata center organizes and manages the metadata in a unified manner to realize the unified loading of metadata, change notifications and data synchronization.
## Data Structure in Metadata Center
The metadata center stores metadata in YAML under the metadata node of the defined namespace and orchestration node, and each logical data source is stored independently.
```
├─orchestration-namespace
│ ├─orchestration-name
│ │ ├──metadata
│ │ │ ├──schema1
│ │ │ │ ├── [YAML text contents]
│ │ │ ├──schema2
│ │ │ │ ├── [YAML text contents]
│ │ │ ├──....
```
## YAML text contents
In the metadata center, metadata is divided into two parts: `configuredSchemaMetaData` and` unconfiguredSchemaMetaDataMap`.
Dynamic modification of metadata content is not supported currently.
```
configuredSchemaMetaData:
tables: # Tables
t_order: # table_name
columns: # Columns
id: # column_name
caseSensitive: false
dataType: 0
generated: false
name: id
primaryKey: trues
order_id:
caseSensitive: false
dataType: 0
generated: false
name: order_id
primaryKey: false
indexs: # Indexes
t_user_order_id_index: # index_name
name: t_user_order_id_index
t_order_item:
columns:
order_id:
caseSensitive: false
dataType: 0
generated: false
name: order_id
primaryKey: false
unconfiguredSchemaMetaDataMap:
ds_0: # DataSources
tables: # Tables
t_user: # table_name
columns: # Columns
user_id: # column_name
caseSensitive: false
dataType: 0
generated: false
name: user_id
primaryKey: false
id:
caseSensitive: false
dataType: 0
generated: false
name: id
primaryKey: true
order_id:
caseSensitive: false
dataType: 0
generated: false
name: order_id
primaryKey: false
indexes: # Indexes
t_user_order_id_index: # index_name
name: t_user_order_id_index
t_user_user_id_index:
name: t_user_user_id_index
primary:
name: PRIMARY
```
### configuredSchemaMetaData
Store metadata for all data sources configured with sharding rules.
### unconfiguredSchemaMetaDataMap
Store metadata for data sources that no sharding rules configured.
## Change Notifications
After DDL is executed through a certain Proxy instance, ShardingSphere stores new metadata in the metadata center first, and then notifies other Proxy instances to synchronize metadata from the metadata center by event broadcast mechanism to ensure metadata consistency.
+++
pre = "<b>3.3.3. </b>"
title = "支持的配置中心/注册中心"
weight = 3
pre = "<b>3.3.4. </b>"
title = "支持的配置/注册/元数据中心"
weight = 4
+++
## SPI
[Service Provider Interface (SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html)是一种为了被第三方实现或扩展的API。它可以用于实现框架扩展或组件替换。
ShardingSphere在数据库治理模块使用SPI方式载入数据到配置中心/注册中心,进行实例熔断和数据库禁用。
ShardingSphere在数据库治理模块使用SPI方式载入数据到配置中心/注册/元数据中心,进行实例熔断和数据库禁用。
目前,ShardingSphere内部支持Zookeeper和etcd这种常用的配置中心/注册中心。
此外,您可以使用其他第三方配置中心/注册中心,并通过SPI的方式注入到ShardingSphere,从而使用该配置中心/注册中心,实现数据库治理功能。
此外,您可以使用其他第三方配置中心/注册/元数据中心,并通过SPI的方式注入到ShardingSphere,从而使用该配置中心/注册/元数据中心,实现数据库治理功能。
## Zookeeper
ShardingSphere官方使用[Apache Curator](http://curator.apache.org/)作为Zookeeper的实现方案(支持配置中心和注册中心)。
ShardingSphere官方使用[Apache Curator](http://curator.apache.org/)作为Zookeeper的实现方案(支持配置中心、注册中心和元数据中心)。
请使用Zookeeper 3.4.6及其以上版本,详情请参见[官方网站](https://zookeeper.apache.org/)
## Etcd
ShardingSphere官方使用[io.etcd/jetcd](https://github.com/etcd-io/jetcd)作为Etcd的实现方案(支持配置中心和注册中心)。
ShardingSphere官方使用[io.etcd/jetcd](https://github.com/etcd-io/jetcd)作为Etcd的实现方案(支持配置中心、注册中心和元数据中心)。
请使用Etcd v3以上版本,详情请参见[官方网站](https://etcd.io/)
## Apollo
......
+++
pre = "<b>3.3.3. </b>"
title = "Available Config Center/Registry Center"
weight = 3
pre = "<b>3.3.4. </b>"
title = "Available Config/Registry/Metadata Center"
weight = 4
+++
## SPI
......@@ -9,15 +9,15 @@ weight = 3
[Service Provider Interface (SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html) is a kind of API that aims to be implemented or extended by the third party.
It can be used to realize framework extension or component replacement.
ShardingSphere uses SPI to load data to the config center/registry center and disable instances and databases. Currently, ShardingSphere supports frequently used registry centers, Zookeeper and Etcd. In addition, by injecting them to ShardingSphere with SPI, users can use other third-party config/registry centers to enable databases orchestration.
ShardingSphere uses SPI to load data to the config center/registry/metadata center and disable instances and databases. Currently, ShardingSphere supports frequently used registry centers, Zookeeper and Etcd. In addition, by injecting them to ShardingSphere with SPI, users can use other third-party config/registry/metadata centers to enable databases orchestration.
## Zookeeper
ShardingSphere adopts [Apache Curator](http://curator.apache.org/) to enable Zookeeper(support config center&registry center). Please use Zookeeper 3.4.6 and above, see the [official website](https://zookeeper.apache.org/) for details.
ShardingSphere adopts [Apache Curator](http://curator.apache.org/) to enable Zookeeper(support config center&registry center&metadata center). Please use Zookeeper 3.4.6 and above, see the [official website](https://zookeeper.apache.org/) for details.
## Etcd
ShardingSphere adopts [io.etcd/jetcd](https://github.com/etcd-io/jetcd) to enable Etcd(support config center&registry center). Please use Etcd v3 and above, see the [official website](https://etcd.io/) for details.
ShardingSphere adopts [io.etcd/jetcd](https://github.com/etcd-io/jetcd) to enable Etcd(support config center&registry center&metadata center). Please use Etcd v3 and above, see the [official website](https://etcd.io/) for details.
## Apollo
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册