未验证 提交 f12cc81e 编写于 作者: L lizhiboo 提交者: GitHub

Init english version of the README

Co-authored-by: N李致波 <lizhibo@lizhibodeMacBook-Pro.local>
上级 cf20a26c
Apache RocketMQ Developer Guide
--------
##### This guide helps develpers understand and use Apache RocketMQ quickly.
### 1. Concepts & Features
- [Concept](Concept.md):introduce basic concepts in RocketMQ.
- [Feature](Feature.md):introduce functional features of RocketMQ's implementations.
### 2. Architecture Design
- [Architecture](architecture.md):introduce RocketMQ's deployment and technical architecture.
- [Design](design.md):introduce design concept of RocketMQ's key mechanisms, including message storage, communication mechanisms, message filter, loadbalance, transaction message, etc.
### 3. Example
- [Example](RocketMQ_Example.md) :introduce RocketMQ's common usage, including basic example, sequence message example, delay message example, batch message example, filter message example, transaction message example, etc.
### 4. Best Practice
- [Best Practice](best_practice.md):introduce RocketMQ's best practice, including producer, consumer, broker, NameServer, configuration of client, and the best parameter configuration of JVM, linux.
- [Message Trace](msg_trace/user_guide.md):introduce how to use RocketMQ's message tracing feature.
- [Auth Management](acl/Operations_ACL.md):introduce how to deployment quickly and how to use RocketMQ cluster enabling auth management feature.
- [Quick Start](dledger/quick_start.md):introduce how to deploy Dledger quickly.
- [Cluster Deployment](dledger/deploy_guide.md):introduce how to deploy Dledger in cluster.
### 5. Operation and maintenance management
- [Operation](operation.md):introduce RocketMQ's deployment modes that including single-master mode, multi-master mode, multi-master multi-slave mode and so on, as well as the usage of operation tool mqadmin.
### 6. API Reference(TODO)
- [DefaultMQProducer API Reference](client/java/API_Reference_DefaultMQProducer.md)
### Examples List
- [basic example](Example_Simple.md)
- [sequence message example](Example_Orderly.md)
- [delay message example](Example_Delay.md)
- [batch message example](Example_Batch.md)
- [filter message example](Example_Filter.md)
- [transaction message example](Example_Transaction.md)
- [openmessaging example](Example_OpenMessaging.md)
# Dledger cluster deployment
---
## preface
This document introduces how to deploy auto failover RocketMQ-on-DLedger Group。
RocketMQ-on-DLedger Group is a broker group with **same name**, needs at least 3 nodes, elect a Leader by Raft algorithm automatically, the others as Follower, replicating data between Leader and Follower for system high available.
RocketMQ-on-DLedger Group can failover automatically, and maintains consistent.
RocketMQ-on-DLedger Group can scale up horizontal, that is, can deploy any RocketMQ-on-DLedger Groups providing services external.
## 1. New cluster deployment
#### 1.1 Write the configuration
each RocketMQ-on-DLedger Group needs at least 3 machines.(assuming 3 in this document)
write 3 configuration files, advising refer to the directory of conf/dledger 's example configuration file.
key configuration items:
| name | meaning | example |
| --- | --- | --- |
| enableDLegerCommitLog | whether enable DLedger  | true |
| dLegerGroup | DLedger Raft Group's name, advising maintain consistent to brokerName | RaftNode00 |
| dLegerPeers | DLedger Group's nodes port infos, each node's configuration stay consistent in the same group. | n0-127.0.0.1:40911;n1-127.0.0.1:40912;n2-127.0.0.1:40913 |
| dLegerSelfId | node id, must belongs to dLegerPeers; each node is unique in the same group. | n0 |
| sendMessageThreadPoolNums | the count of sending thread, advising set equal to the cpu cores. | 16 |
the following presents an example configuration conf/dledger/broker-n0.conf.
```
brokerClusterName = RaftCluster
brokerName=RaftNode00
listenPort=30911
namesrvAddr=127.0.0.1:9876
storePathRootDir=/tmp/rmqstore/node00
storePathCommitLog=/tmp/rmqstore/node00/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n0-127.0.0.1:40911;n1-127.0.0.1:40912;n2-127.0.0.1:40913
## must be unique
dLegerSelfId=n0
sendMessageThreadPoolNums=16
```
### 1.2 Start Broker
Startup stays consistent with the old version.
`nohup sh bin/mqbroker -c conf/dledger/xxx-n0.conf & `
`nohup sh bin/mqbroker -c conf/dledger/xxx-n1.conf & `
`nohup sh bin/mqbroker -c conf/dledger/xxx-n2.conf & `
## 2. Upgrade old cluster
If old cluster deployed in Master mode, then each Master needs to be transformed into a RocketMQ-on-DLedger Group.
If old cluster deployed in Master-Slave mode, then each Master-Slave group needs to be transformed into a RocketMQ-on-DLedger Group.
### 2.1 Kill old Broker
execute kill command, or call `bin/mqshutdown broker`.
### 2.2 Check old Commitlog
Each node in RocketMQ-on-DLedger group is compatible with old Commitlog, but Raft replicating process works on the adding message only. So, to avoid occurring exceptions, old Commitlog must be consistent.
If old cluster deployed in Master-Slave mode, it maybe inconsistent after shutdown. Advising use md5sum to check at least 2 recently Commitlog file, if occur inconsistent, maintain consistent by copy.
Although RocketMQ-on-DLedger Group can deployed with 2 nodes, it lacks failover ability(at least 3 nodes can tolerate one node fail).
Make sure that both Master and Slave's Commitlog is consistent, then prepare 3 machines, copy old Commitlog from Master to this 3 machines(BTW, copy the config directory).
Then, go ahead to set configurations.
### 2.3 Modify configuration
Refer to New cluster deployment.
### 2.4 Restart Broker
Refer to New cluster deployment.
# Dledger Quick Deployment
---
### preface
This document is mainly introduced for how to build and deploy auto failover RocketMQ cluster based on DLedger.
For detailed new cluster deployment and old cluster upgrade document, please refer to [Deployment Guide](deploy_guide.md)
### 1. Build from source code
Build phase contains two parts, first, build DLedger, then build RocketMQ.
#### 1.1 Build DLedger
`git clone https://github.com/openmessaging/openmessaging-storage-dledger.git`
`cd openmessaging-storage-dledger`
`mvn clean install -DskipTests`
#### 1.2 Build RocketMQ
`git clone https://github.com/apache/rocketmq.git`
`cd rocketmq`
`git checkout -b store_with_dledger origin/store_with_dledger`
`mvn -Prelease-all -DskipTests clean install -U`
### 2. Quick Deployment
after build successful
`cd distribution/target/apache-rocketmq`
`sh bin/dledger/fast-try.sh start`
if the above commands executed successfully, then check cluster status by using mqadmin operation commands.
`sh bin/mqadmin clusterList -n 127.0.0.1:9876`
If everything goes well, the following content will appear:
![ClusterList](https://img.alicdn.com/5476e8b07b923/TB11Z.ZyCzqK1RjSZFLXXcn2XXa)
(BID is 0 indicate Master,the others are Follower)
After startup successful, producer can produce message, and then test failover scenario.
Stop cluster fastly, execute the following command:
`sh bin/dledger/fast-try.sh stop`
Quick deployment, default configuration is in directory conf/dledger,default storage path is /tmp/rmqstore.
### 3. Failover
After successful deployment, kill Leader process(as the above example, kill process that binds port 30931), about 10 seconds elapses, use clusterList command check cluster's status, Leader switch to another node.
# message trace
----
## 1. Message trace data's key properties
| Producer End| Consumer End| Broker End|
| --- | --- | --- |
| produce message | consume message | message's topic |
| send message time | delivery time, delivery rounds  | message store location |
| whether the message was sent successfully | whether message was consumed successfully | message's key |
| send cost-time | consume cost-time | message's tag value |
## 2. Enable message trace in cluster deployment
### 2.1 Broker's configuration file
following by Broker's properties file configuration that enable message trace:
```
brokerClusterName=DefaultCluster
brokerName=broker-a
brokerId=0
deleteWhen=04
fileReservedTime=48
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
storePathRootDir=/data/rocketmq/rootdir-a-m
storePathCommitLog=/data/rocketmq/commitlog-a-m
autoCreateSubscriptionGroup=true
## if msg tracing is open,the flag will be true
traceTopicEnable=true
listenPort=10911
brokerIP1=XX.XX.XX.XX1
namesrvAddr=XX.XX.XX.XX:9876
```
### 2.2 Common mode
Each Broker node in RocketMQ cluster used for storing message trace data that client collected and sent. So, there is no requirements and limitations to the size of Broker node in RocketMQ cluster.
### 2.3 IO physical isolation mode
For huge amounts of message trace data scenario, we can select any one Broker node in RocketMQ cluster used for storing message trace data special, thus, common message data's IO are isolated from message trace data's IO in physical, not impact each other. In this mode, RocketMQ cluster must have at least two Broker nodes, the one that defined as storing message trace data.
### 2.4 Start Broker that enable message trace
`nohup sh mqbroker -c ../conf/2m-noslave/broker-a.properties &`
## 3. Save the definition of topic that with support message trace
RocketMQ's message trace feature supports two types of storage.
### 3.1 System level TraceTopic
Be default, message trace data is stored in system level TraceTopic(topic name: **RMQ_SYS_TRACE_TOPIC**). That topic will be created at startup of broker(As mentioned above, set **traceTopicEnable** to **true** in Broker's configuration).
### 3.2 User defined TraceTopic
If user don't want to store message trace data in system level TraceTopic, he can create user defined TraceTopic used for storing message trace data(that is, create common topic for storing message trace data). The following part will introduce how client SDK support user defined TraceTopic.
## 4. Client SDK demo with message trace feature
For business system adapting to use RocketMQ's message trace feature easily, in design phase, the author add a switch parameter(**enableMsgTrace**) for enable message trace; add a custom parameter(**customizedTraceTopic**) for user defined TraceTopic.
### 4.1 Enable message trace when sending messages
```
DefaultMQProducer producer = new DefaultMQProducer("ProducerGroupName",true);
producer.setNamesrvAddr("XX.XX.XX.XX1");
producer.start();
try {
{
Message msg = new Message("TopicTest",
"TagA",
"OrderID188",
"Hello world".getBytes(RemotingHelper.DEFAULT_CHARSET));
SendResult sendResult = producer.send(msg);
System.out.printf("%s%n", sendResult);
}
} catch (Exception e) {
e.printStackTrace();
}
```
### 4.2 Enable message trace when subscribe messages
```
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("CID_JODIE_1",true);
consumer.subscribe("TopicTest", "*");
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
consumer.setConsumeTimestamp("20181109221800");
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
consumer.start();
System.out.printf("Consumer Started.%n");
```
### 4.3 Self-defined topic support message trace
Adjusting instantiation of DefaultMQProducer and DefaultMQPushConsumer as following code to support user defined TraceTopic.
```
##Topic_test11111 should be created by user, used for storing message trace data.
DefaultMQProducer producer = new DefaultMQProducer("ProducerGroupName",true,"Topic_test11111");
......
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("CID_JODIE_1",true,"Topic_test11111");
......
```
此差异已折叠。
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册