From 7f53f64a0d49f5c693fae4b453650fe8661ad3c9 Mon Sep 17 00:00:00 2001 From: He Wang Date: Thu, 17 Nov 2022 18:27:09 +0800 Subject: [PATCH] [docs] Update README and build doc in English (#40) * update README * update manual * revert manual file name --- README.md | 159 +++++++++++++++++++++---------------------------- docs/build.md | 64 ++++++++++++++++++++ docs/manual.md | 139 +----------------------------------------- 3 files changed, 132 insertions(+), 230 deletions(-) create mode 100644 docs/build.md diff --git a/README.md b/README.md index a91e113..a28e3b5 100644 --- a/README.md +++ b/README.md @@ -1,127 +1,102 @@ -# OceanBase Migration Serivce LogProxy +# OceanBase LogProxy -OceanBase增量日志代理服务，是 [OMS](https://www.oceanbase.com/product/oms) 的一部分。基于 [liboblog](https://github.com/oceanbase/oceanbase), 以服务的形式，提供实时增量链路接入和管理能力，方便应用接入OceanBase增量日志；能够解决网络隔离的情况下，订阅增量日志的需求；并提供多种链路接入方式： - - [oblogclient](https://github.com/oceanbase/oblogclient) - - Canal - - More... +OceanBase LogProxy (CE) is a proxy service of [OceanBase CE](https://github.com/oceanbase/oceanbase). It is a part of [OMS](https://www.oceanbase.com/product/oms), and it can establish and manage connections with OceanBase for incremental log reading even with a isolated network. -## Quick start +## Instructions before use + +### Version compatibility + +LogProxy is based on [libobcdc](https://github.com/oceanbase/oceanbase/tree/master/src/logservice/libobcdc) (former `liboblog`), so you should install the corresponding version of it firstly. The libobcdc is packaged in `oceanbase-ce-devel` before 4.0.0, and is packaged in `oceanbase-ce-cdc` in 4.0.0 and the later version, both of which can be found in the [official download page](https://open.oceanbase.com/softwareCenter/community) or [official mirror](https://mirrors.aliyun.com/oceanbase/community/stable/el/). + +| libobcdc | oblogproxy | +|----------|------------| +| 3.1.1 | 1.0.0 | +| 3.1.2 | 1.0.1 | +| 3.1.3 | 1.0.2 | +| 3.1.4 | 1.0.3 | +| 4.0.0 | 1.1.0 | + +### Installation + +LogProxy service doesn't need params about OceanBase cluster to get started, one LogProxy can subscribe to multiple OceanBase clusters at the same time, and the connection configuration is passed from the client. + +LogProxy will use a lot of memory, so it is strongly recommended to deploy it separately from the OceanBase server. + +## Getting started + +### Install + +You can install a released version of LogProxy or build it from the source code. + +#### Install a released version + +If you want to install a released version, firstly you need to configure the yum repo. -### 1. 配置OceanBase社区版yum源 ```bash yum install -y yum-utils yum-config-manager --add-repo https://mirrors.aliyun.com/oceanbase/OceanBase.repo ``` -### 2. 下载预编译包 +Then you can install the rpm file by one of the following way: + + Download from [release page](https://github.com/oceanbase/oblogproxy/releases), [official download page](https://open.oceanbase.com/softwareCenter/community) or [official mirror](https://mirrors.aliyun.com/oceanbase/community/stable/el/), and install it with `yum install -y oblogproxy-{version}.{arch}.rpm` + + Install it with `yum install -y oblogproxy-{version}` -预编译的产出在：[Release](http://mirrors.aliyun.com/oceanbase/community/stable/el/7/x86_64/) ，oblogproxy的包名是"oblogproxy-xxxx.系统版本.x86_64.rpm"，根据自己的系统选取，安装： -```bash -yum install -y oblogproxy-xxxx.系统版本.x86_64.rpm -``` +The installation directory is `/usr/local/oblogproxy` by default. + +#### Build from source code + +See [How to build](docs/build.md). -oblogproxy会安装在目录 `/usr/local/oblogproxy` 。 +### Configure -需要注意的是，oblogproxy 对 [obcdc](https://github.com/oceanbase/oceanbase/tree/master/src/logservice/libobcdc) (原liboblog) 有版本依赖，当通过 oceanbase-ce-devel 引入该依赖时，默认的安装流程可能会安装最新版，因此建议先安装指定版本的 oceanbase-ce-devel，版本对应关系： +For security reasons, LogProxy needs to configure the username and password of a certain user, which must be a sys tenant user of the OceanBase to connect with. Note that the username here should not contain cluster name or tenant name. -| obcdc (devel) | oblogproxy | -|---------------|------------| -| 3.1.1 | 1.0.0 | -| 3.1.2 | 1.0.1 | -| 3.1.3 | 1.0.2 | -| 3.1.4 | 1.0.3 | -| 4.0 | 1.1.0 | +You can configure the username and password by one of the following ways: +- Add it to local conf at `conf/conf.json`. +- Set it in the client params. See the [client doc](https://github.com/oceanbase/oblogclient/blob/master/docs/quickstart/logproxy-client-tutorial.md) for details. + +#### Add it to local conf + +Firstly, get the encrypted username and password. -### 3. 配置系统租户 -获得observer的sys租户账号密码，通常在创建observer集群时创建，也可以单独创建。oblogproxy需要加密的配置，执行以下命令即可得到： ```bash -# 这里假设账号密码分别为：user，pswd -./logproxy -x user -# 会输出 4B9C75F64934174F4E77EE0E9A588118 -./logproxy -x pswd -# 会输出 DCE2AF09D006D6A440816880B938E7B3 -``` -把获得账号密码密文分别配置到`/usr/local/oblogproxy/conf/conf.json`中的`ob_sys_username`和`ob_sys_password`字段，例如： -```json -{ - "ob_sys_username": "4B9C75F64934174F4E77EE0E9A588118", - "ob_sys_password": "DCE2AF09D006D6A440816880B938E7B3" -} +./bin/logproxy -x username +./bin/logproxy -x password ``` -### 4. 运行 +Then add the outputs to `ob_sys_username` and `ob_sys_password` at `conf/conf.json`. + +### Start + +You can start the service by the following command. + ```bash -cd /usr/local/oblogproxy bash ./run.sh start ``` -### 5. 用oblogclient订阅 -此时可以使用 [oblogclient](https://github.com/oceanbase/oblogclient) 进行OB数据订阅。 Maven依赖，见： [Maven Repo](https://search.maven.org/search?q=g:com.oceanbase.logclient) -```xml - - com.oceanbase.logclient - logproxy-client - 1.0.1 - -``` -编写代码，参考： -```Java -ObReaderConfig config = new ObReaderConfig(); -// 设置OceanBase root server 地址列表，格式为（可以支持多个，用';'分隔）：ip1:rpc_port1:sql_port1;ip2:rpc_port2:sql_port2 -config.setRsList("127.0.0.1:2882:2881;127.0.0.2:2882:2881"); -// 设置用户名和密码（非系统租户） -config.setUsername("root"); -config.setPassword("root"); -// 设置启动位点（UNIX时间戳，单位s）, 0表示从当前时间启动。 -config.setStartTimestamp(0L); -// 设置订阅表白名单，格式为：tenant.db.table, '*'表示通配. -config.setTableWhiteList("sys.*.*"); - -// 指定oblogproxy服务地址，创建实例. -LogProxyClient client = new LogProxyClient("127.0.0.1", 2983, config); -// 添加 RecordListener -client.addListener(new RecordListener() { - @Override - public void notify(LogMessage message){ - // 处理消息 - } - - @Override - public void onException(LogProxyClientException e) { - // 处理错误 - if (e.needStop()) { - // 不可恢复异常，需要停止Client - client.stop(); - } - } -}); - -// 启动 -client.start(); -client.join(); -``` +You can also start LogProxy with customized libobcdc by executing the following command. -更多详情见 [使用文档](https://github.com/oceanbase/oblogclient) +```bash +export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/libobcdc +bash ./run.sh start +``` -## 注意 -- 配置里面无需指定具体的OB集群信息，理论上oblogproxy能同时订阅多个OB集群，只要与所有observer网络通，且OB集群包含配置中的sys租户账号密码。（需要配置sys租户账号密码由于安全原因，通常不对oblogclient侧的用户暴露） -- 本身是无状态，订阅哪个OB，哪个库表都由oblogclient传入，且增量链路的位点等信息也需要oblogclient端自行保存。断开重连后，对于oblogproxy来讲，是创建全新的链路。 -- 消耗内存较多，强烈建议和observer分开部署。 +Then you can use [oblogclient](https://github.com/oceanbase/oblogclient) to subscribe the log data from LogProxy, and the service is bind to port `2983` by default. -## 文档 -- [编译](./docs/manual.md#编译) -- [运行](./docs/manual.md#运行) -- [配置](./docs/manual.md#配置) +The service log of LogProxy is located at `logs/`, and the service log of LogReader (task thread) is located at `run/{client-id}/logs/`. ## Licencing + OceanBase Database is under MulanPubL - 2.0 license. You can freely copy and use the source code. When you modify or distribute the source code, please obey the MulanPubL - 2.0 license. ## Contributing + Contributions are warmly welcomed and greatly appreciated. Here are a few ways you can contribute: - Raise us an [issue](https://github.com/oceanbase/oblogproxy/issues). - Submit Pull Requests. ## Support -In case you have any problems when using OceanBase Database, welcome reach out for help: + +In case you have any problems when using OceanBase LogProxy, welcome reach out for help: - [GitHub Issue](https://github.com/oceanbase/oblogproxy/issues) - [Official Website](https://open.oceanbase.com/) diff --git a/docs/build.md b/docs/build.md new file mode 100644 index 0000000..f2bfedd --- /dev/null +++ b/docs/build.md @@ -0,0 +1,64 @@ +## How to Build + +### Preparation + +#### Install CMake + +Install [CMake](https://cmake.org/download) 3.20 or later. + +#### Install Dependencies + +Fedora based (including CentOS, Fedora, OpenAnolis, RedHat, UOS, etc.) + +```bash +yum install which git wget rpm rpm-build cpio gcc gcc-c++ make glibc-devel glibc-headers libstdc++-static binutils openssl-devel libaio-devel +``` + +Debian based (including Debian, Ubuntu, etc.) + +```bash +apt-get install git wget rpm rpm2cpio cpio gcc make build-essential binutils +``` + +### Build + +Get the source code and execute the following commands in the project directory. + +```bash +mkdir buildenv && cd buildenv +cmake .. +make -j 6 +``` + +Then there should be a binary output `logproxy` in the current directory. You can set the working directory by the following commands. + +```bash +mkdir -p ./oblogproxy/bin ./oblogproxy/run +cp -r ../conf ./oblogproxy/ +cp ../script/run.sh ./oblogproxy/ +cp logproxy ./oblogproxy/bin/ +``` + +### Build Options + +There are some build options. + +| Option | Default | Description | +|---------------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------| +| WITH_DEBUG | ON | Debug mode flag. | +| WITH_ASAN | OFF | Flag of whether to build with [AddressSanitizer](https://github.com/google/sanitizers). | +| WITH_TEST | OFF | Flag of whether to build test. | +| WITH_DEMO | OFF | Flag of whether to build demo. | +| WITH_GLOG | ON | Flag of whether to build with glog. | +| WITH_DEPS | ON | Flag of whether to automatically download precompiled dependencies. | +| USE_LIBOBLOG | OFF | Flag of whether to build with a customized precompiled libobcdc/liboblog. | +| USE_OBCDC_NS | ON | Flag of whether to build with libobcdc, use 'OFF' to build with former liboblog (before 3.1.3). | +| USE_CXX11_ABI | ON | Flag of whether to build with C++11 ABI. Note that if precompiled dependencies are used, they need to be consistent, otherwise the symbols will not be found | + +For example, if you want to build with precompiled libobcdc, you can use the following commands. + +```bash +mkdir buildenv && cd buildenv +CMAKE_INCLUDE_PATH=/path/to/libobcdc CMAKE_LIBRARY_PATH=/path/to/libobcdc cmake -DUSE_LIBOBLOG=ON .. +make -j 6 +``` \ No newline at end of file diff --git a/docs/manual.md b/docs/manual.md index c830a10..86c45b9 100644 --- a/docs/manual.md +++ b/docs/manual.md @@ -1,147 +1,10 @@ -# 参考手册 - -## 编译 - -### 前置条件 - -#### 1. 安装CMake -要求版本大于等于3.20, 安装包见：[https://cmake.org/download](https://cmake.org/download) - -#### 2. 安装依赖 - -Fedora based (including CentOS, Fedora, OpenAnolis, RedHat, UOS, etc.) -```bash -yum install which git wget rpm rpm-build cpio gcc gcc-c++ make glibc-devel glibc-headers libstdc++-static binutils openssl-devel libaio-devel -``` - -Debian based (including Debian, Ubuntu, etc.) -```bash -apt-get install git wget rpm rpm2cpio cpio gcc make build-essential binutils -``` - -### 编译操作 - -#### 执行CMake编译 -```shell -mkdir buildenv && cd buildenv -# 生成 -cmake .. -# 执行 -make -j 6 -``` - -一切正常的话，在当前目录产出了`logproxy`二进制文件。 - -### 编译选项 -上述编译过程，可以添加编译选项改变默认的行为。例如, 编译出Demo。成功后，当前目录还会产出`demo_client`二进制。 -```shell -mkdir buildenv && cd buildenv -cmake -DWITH_DEMO=ON .. -make -j 6 -``` - -#### 自定义 liboblog.so -您可以自己指定编译所使用的liboblog，下载预编译的包或者自行编译二选一。 - -**下载预编译包：** - -预编译的产出在：[Release](https://github.com/oceanbase/oceanbase/releases) ，liboblog的包名是"oceanbase-ce-devel-xxxx.系统版本.x86_64.rpm"，根据自己的系统选取，解压： -```bash -rpm2cpio oceanbase-ce-devel-xxxx.系统版本.x86_64.rpm | cpio -div && mv ./usr ./liboblog -``` -解压后： -- liboblog.so: 在 `./liboblog/lib` 目录下 -- liboblog.h: 在 `./liboblog/include` 目录下 - -**编译liboblog：** - -下面参考了 [liboblog编译说明](https://open.oceanbase.com/docs/community/oceanbase-database/V3.1.1/abyu9b) ，编译 [OceanBase社区版](https://github.com/oceanbase/oceanbase) 时，添加参数 OB_BUILD_LIBOBLOG=ON。 -```bash -git clone https://github.com/oceanbase/oceanbase.git && cd oceanbase -bash ./build.sh release --init -DOB_BUILD_LIBOBLOG=ON --make -``` - -获得编译产出： -- liboblog.so: 在 `build_release/src/liboblog/src/` 目录下。 -- liboblog.h: 在 `src/liboblog/src/` 目录下。 - -**编译oblogproxy：** - -这里假设您把`liboblog.so`和`liboblog.h`都放在了`/path/to/liboblog`。 - -需要在编译命令中打开自定义liboblog的开关并指定路径： -```shell -mkdir buildenv && cd buildenv -# 设置CMake环境变量从而可以找到预编译的liboblog -CMAKE_INCLUDE_PATH=/path/to/liboblog CMAKE_LIBRARY_PATH=/path/to/liboblog cmake -DUSE_LIBOBLOG=ON .. -# 执行 -make -j 6 -``` - -#### 全部编译参数 - -| 选项 | 默认 | 说明 | -| ------ |-----|-------------------------------------------------------------------------------------------| -| WITH_DEBUG | ON | 调试模式带 Debug 符号 | -| WITH_ASAN | OFF | 编译带 [AddressSanitizer](https://github.com/google/sanitizers) | -| WITH_TEST | OFF | 测试 | -| WITH_DEMO | OFF | Demo | -| WITH_GLOG | ON | 使用glog | -| WITH_DEPS | ON | 自动下载预编译依赖 | -| USE_LIBOBLOG | OFF | 使用自定义预编译的liboblog | -| USE_OBCDC_NS | ON | 是否使用obcdc进行编译。注意oblogproxy 对 obcdc (原liboblog) 有版本依赖,USE_OBCDC_NS=OFF时，兼容obcdc 3.1.2及之前版本 | -| USE_CXX11_ABI | ON | 是否使用C++11 ABI。注意如果用了预编译的依赖，需要保持一致，否则会找不到符号 | - -### 编译依赖说明 +## 依赖说明 默认情况下，会自动下载并编译依赖库。有几个点这里说明下： - **openssl**：当前使用的版本是：1.0.1e，1.0.*和1.1.*版本API少量不兼容，当前liboblog和logproxy都是是基于1.0.*实现的，需要版本一致。 - **liboblog**：如前文描述，您也可以自行获取或编译，并由环境变量指定路径，运行时需要指定LD_LIBRARY_PATH。 - **libaio**：liboblog依赖。 -## 运行 -oblogproxy 单一配置文件，用代码目录下的 `conf.json`。 - -### 1. 配置系统租户 -获得observer的sys租户账号密码，通常在创建observer集群时创建，也可以单独创建。oblogproxy需要加密的配置，执行以下命令即可得到： -```bash -# 这里假设账号密码分别为：user，pswd -./logproxy -x user -# 会输出 4B9C75F64934174F4E77EE0E9A588118 -./logproxy -x pswd -# 会输出 DCE2AF09D006D6A440816880B938E7B3 -``` -把获得账号密码密文分别配置到`conf.json`中的`ob_sys_username`和`ob_sys_password`字段，例如： -```json -{ - "ob_sys_username": "4B9C75F64934174F4E77EE0E9A588118", - "ob_sys_password": "DCE2AF09D006D6A440816880B938E7B3" -} -``` - -### 2. 组织程序目录 -```bash -# 创建程序目录 -mkdir -p ./oblogproxy/bin ./oblogproxy/run -# 复制配置文件 -cp -r ../conf ./oblogproxy/ -# 复制起停脚本 -cp ../script/run.sh ./oblogproxy/ -# 复制程序二进制 -cp logproxy ./oblogproxy/bin/ -``` - - -### 3. 启动oblogproxy -```shell -cd ./oblogproxy -# 指定liboblog.so目录，让oblogproxy可以动态依赖 -export LD_LIBRARY_PATH=/path/to/liboblog -bash ./run.sh start -``` -默认监听`2983`端口，修改`conf.json`中的`service_port`字段可更换监听端口。 - -此时可以使用 [oblogclient](https://github.com/oceanbase/oblogclient) 进行OB数据订阅，见 [使用文档](https://github.com/oceanbase/oblogclient) ## 链路加密 ### oblogclient与oblogproxy间TLS通信 -- GitLab