未验证 提交 ca097502 编写于 作者: H He Wang 提交者: GitHub

[doc] update usage and add logmessage document (#18)

* add docs

* fix javadoc warning
上级 2be57fb0
OceanBase Log Client
--------------------
OceanBase Log Client is a library for obtaining log of [OceanBase](https://github.com/oceanbase/oceanbase).
Getting Started
---------------
OceanBase Log Client is a library for obtaining log of [OceanBase](https://github.com/oceanbase/oceanbase). There are modules as following:
There are modules as following:
- `common`: some common utils
- `logproxy-client`: the client for [oblogproxy](https://github.com/oceanbase/oblogproxy)
You can try the `logproxy-client` following the [LogProxyClient Tutorial](docs/quickstart/logproxy-client-tutorial.md).
Communication
---------------
* [Official Q&A Website (Chinese)](https://open.oceanbase.com/answer) (Q&A, Ideas, General discussion)
* [Official Q&A Website (Chinese)](https://open.oceanbase.com/answer) (Q&A, ideas, general discussion)
* [GitHub Issues](https://github.com/oceanbase/oblogclient/issues) (Bug reports, feature requests)
* DingTalk Group (chat): 33254054
Binaries/Download
----------------
Binaries and dependency information for Maven, Ivy, Gradle and others can be found at http://search.maven.org.
Releases are available in the Maven Central repository. Take also a look at the [Releases](https://github.com/oceanbase/oblogclient/releases).
Example for Maven:
```xml
<dependency>
<groupId>com.oceanbase.logclient</groupId>
<artifactId>logproxy-client</artifactId>
<version>x.y.z</version>
</dependency>
```
If you'd rather like the latest snapshots of the upcoming major version, use our Maven snapshot repository and declare the appropriate dependency version.
```xml
<dependency>
<groupId>com.oceanbase.logclient</groupId>
<artifactId>logproxy-client</artifactId>
<version>x.y.z-SNAPSHOT</version>
</dependency>
<repositories>
<repository>
<id>sonatype-snapshots</id>
<name>Sonatype Snapshot Repository</name>
<url>https://s01.oss.sonatype.org/content/repositories/snapshots/</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
```
Usage
-----------
Basic usage
```java
ObReaderConfig config = new ObReaderConfig();
// Set root server list in format [ip1:rpc_port1:sql_port1;ip2:rpc_port2:sql_port2],
// while multiple servers are seperated by ';'.
config.setRsList("127.0.0.1:2882:2881");
// Set username and password.
config.setUsername("root@sys");
config.setPassword("root@sys");
// Set timestamp of start point in seconds, and zero means starting from now.
config.setStartTimestamp(0L);
// Set table whitelist in format [tenant.db.table], and '*' indicates any value.
config.setTableWhiteList("sys.*.*");
// Create a client instance.
LogProxyClient client = new LogProxyClient("127.0.0.1", 2983, config);
// Add a RecordListener to handle log messages.
client.addListener(new RecordListener() {
@Override
public void notify(LogMessage message){
// process
}
@Override
public void onException(LogProxyClientException e) {
if (e.needStop()) {
// handle error and stop client
client.stop();
}
}
});
// Start and wait the Netty channel.
client.start();
client.join();
```
Use [SslContext](https://netty.io/4.1/api/io/netty/handler/ssl/SslContext.html) to encrypt communication between log client and log proxy.
```java
SslContext sslContext = SslContextBuilder.forClient()
.sslProvider(SslContext.defaultClientProvider())
.trustManager(this.getClass().getClassLoader().getResourceAsStream("server.crt"))
.keyManager(this.getClass().getClassLoader().getResourceAsStream("client.crt"),
this.getClass().getClassLoader().getResourceAsStream("client.key"))
.build();
LogProxyClient client = new LogProxyClient("127.0.0.1", 2983, config, sslContext);
```
Once the client is successfully started, you should be able to receive `HEARTBEAT` and other messages in the notify method.
* DingTalk Group (Q&A, general discussion): 33254054
License
-------
......
......@@ -879,7 +879,8 @@ public class DataMessage extends Message {
* Construct the message from DataInputStream.
*
* @param reader is the DataInputStream.
* @throws IOException
* @param regionId is the region id.
* @throws IOException if an IOException occurs.
*/
public void mergeFrom(final DataInputStream reader, String regionId) throws IOException {
do {
......
# LogMessage
LogMessage is a struct to store log messages, see the [class file](../../common/src/main/java/com/oceanbase/oms/logmessage/LogMessage.java) for its definition.
## LogMessage Struct
A LogMessage object mainly has the following fields (getter):
- *RawData*: Byte array that contains all details of the log message.
- *DbType*: Type of source database, here we only use `OCEANBASE1`, which means OceanBase 1.0 or higher version.
- *Opt*: Operation type, here should be one of `BEGIN`, `COMMIT`, `INSERT`, `UPDATE`, `DELETE`, `DDL`, `HEARTBEAT`.
- *DbName*: Database name, here it is in format of `tenant_name.database_name`.
- *TableName*: Table name.
- *Timestamp*: Timestamp in seconds.
- *OB10UniqueId*: Transaction id (string) of log message, only recorded in `BEGIN` or DML (`INSERT`, `UPDATE`, `DELETE`).
- *FieldList*: A list of row fields.
## Field List in LogMessage
The item in *FieldList* of LogMessage is of type `DataMessage.Record.Field`, and one Field corresponding to a column of one row. A Field struct mainly contains fields as following:
- *length*: The length of `value` field.
- *primaryKey*: Flag of whether the column is the primary key.
- *name*: Column name.
- *type*: Type of the column, raw value is the const in `LogMessageTypeCode`.
- *flag*: Flag of whether the Field is generated/parsed.
- *encoding*: Encoding of the column.
- *value*: Column value.
- *prev*: Flag of whether the Field is the old value of the column.
Note that the Field struct here contains the type information, which is different from MySQL binlog. The value of a Field is of `ByteString` type, which could be used as a byte array or a string, both of which can easily cast to other types.
The content of Field list in the LogMessage is related to the operation type:
- `BEGIN``COMMIT``HEARTBEAT`:null
- `DDL`: One Field with ddl sql in value field.
- `INSERT`: The column value list of the new row.
- `UPDATE`: Both the old and new column values of the row. The list should be [field_0_old, field_0_new, field_1_old, field_1_new, ...].
- `DELETE`: The column value list of the old row.
## Usage
You can see which projects use `logproxy-client` [here](https://github.com/oceanbase/oblogclient/network/dependents?package_id=UGFja2FnZS0yODMzMjE5Nzc1).
# LogProxyClient Tutorial
[oblogproxy](https://github.com/oceanbase/oblogproxy) (OceanBase Log Proxy, hereinafter LogProxy) is a proxy service which can fetch incremental log data from OceanBase. This tutorial will show you how to use LogProxyClient to connect to LogProxy and get the log data.
## Preparation
There are some requirements to use LogProxyClient:
1. JDK 1.8 or higher version installed.
2. LogProxy started.
3. SSL certificate files if SSL encryption is enabled.
4. Maven or Gradle installed, otherwise you need download the jar file manually.
## Binaries/Download
Releases are available in the [Maven Central](https://mvnrepository.com/artifact/com.oceanbase.logclient/logproxy-client), you can also download the jar file manually from the [archive](https://repo1.maven.org/maven2/com/oceanbase/logclient/logproxy-client/).
Here is an example for Maven:
```xml
<dependency>
<groupId>com.oceanbase.logclient</groupId>
<artifactId>logproxy-client</artifactId>
<version>x.y.z</version>
</dependency>
```
If you'd rather like the latest snapshots of the upcoming major version, use our Maven snapshot repository and declare the appropriate dependency version.
```xml
<dependency>
<groupId>com.oceanbase.logclient</groupId>
<artifactId>logproxy-client</artifactId>
<version>x.y.z-SNAPSHOT</version>
</dependency>
<repositories>
<repository>
<id>sonatype-snapshots</id>
<name>Sonatype Snapshot Repository</name>
<url>https://s01.oss.sonatype.org/content/repositories/snapshots/</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
```
## Usage
### Basic Usage
To connect to LogProxy, there are some parameters to set in `ObReaderConfig`:
- *rootserver_list*: Root server list of OceanBase cluster in format `ip1:rpc_port1:sql_port1;ip2:rpc_port2:sql_port2`, IP address here must be able to be resolved by LogProxy.
- *cluster_username*: Username for OceanBase, the format is `username@tenant_name#cluster_name` when connecting to [obproxy](https://github.com/oceanbase/obproxy) or `username@tenant_name` when directly connecting to OceanBase server.
- *cluster_password*: Password for OceanBase when using configured `cluster_username`.
- *first_start_timestamp*: Start timestamp in seconds, and zero means starting from now.
- *tb_white_list*: Table whitelist in format `tenant_name.database_name.table_name`, and `*` indicates any value.
Here is an example to set ObReaderConfig with a user of sys tenant, and the OceanBase and LogProxy server are on the same machine.
```java
ObReaderConfig config = new ObReaderConfig();
config.setRsList("127.0.0.1:2882:2881");
config.setUsername("user@sys");
config.setPassword("pswd");
config.setStartTimestamp(0L);
config.setTableWhiteList("sys.*.*");
```
Once ObReaderConfig is set properly, you can use it to instance a LogProxyClient and listen to the log data.
```java
LogProxyClient client = new LogProxyClient("127.0.0.1", 2983, config);
// Add a RecordListener to handle log messages.
client.addListener(new RecordListener() {
@Override
public void notify(LogMessage message){
// add process here
}
@Override
public void onException(LogProxyClientException e) {
if (e.needStop()) {
// add error hander here
client.stop();
}
}
});
client.start();
client.join();
```
The method `LogProxyClient.start()` will start a new thread which serving with a netty socket to receive data from LogProxy.
For details about `LogMessage`, see [LogMessage](../formats/logmessage.md).
### SSL Encryption
If SSL verification is enabled at LogProxy, you should instance a LogProxyClient with [SslContext](https://netty.io/4.1/api/io/netty/handler/ssl/SslContext.html). For example:
```java
SslContext sslContext = SslContextBuilder.forClient()
.sslProvider(SslContext.defaultClientProvider())
.trustManager(this.getClass().getClassLoader().getResourceAsStream("ca.crt"))
.keyManager(
this.getClass().getClassLoader().getResourceAsStream("client.crt"),
this.getClass().getClassLoader().getResourceAsStream("client.key"))
.build();
LogProxyClient client = new LogProxyClient("127.0.0.1", 2983, config, sslContext);
```
Here you need provide following files:
- *ca.crt*: Trusted certificate for verifying the remote endpoint's certificate, should be same with LogProxy.
- *client.crt*: Certificate of this client.
- *client.key*: Private key of this client.
See [manual](https://github.com/oceanbase/oblogproxy/blob/master/docs/manual.md) of LogProxy for more details about SSL encryption.
## Heartbeat and Troubleshooting
Once the connection is established properly, LogProxy will start to fetch log messages from OceanBase and send them to LogProxyClient. When the connection is idle, LogProxy will send heartbeat messages to LogProxyClient.
Note that when LogProxy receives the ObReaderConfig, it will only check the format but won't verify the content. So only when the LogProxyClient receives log messages or heartbeat messages can we be sure that the connection is established properly. If the connection doesn't work properly and there is no error message in log of LogProxyClient, you need to check the log of `oblogreader` at LogProxy side, which is under `$(logproxy_home)/run` by default.
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册