Skip to content

R
Rocketmq

Apache RocketMQ / Rocketmq
上一次同步 大约 3 年

通知 267
Star 16139
Fork 68
R
Rocketmq
未验证 提交 70295453 编写于 作者: L lizhiboo 提交者： GitHub
浏览文件
操作

Merge branch 'apache:develop' into develop

上级 6cee569f 9a5ec908
展开全部 隐藏空白更改
内联 并排
Showing with 6580 addition and 1240 deletion
.github/PULL_REQUEST_TEMPLATE.md
浏览文件 @ 70295453
**Make sure set the target branch to `develop`**
## What is the purpose of the change
XXXXX
......
.travis.yml
浏览文件 @ 70295453
dist: trusty
dist: bionic
notifications:
email:
......@@ -9,22 +9,38 @@ notifications:
language: java
jdk:
- oraclejdk8
matrix:
include:
# On OSX, run with default JDK only.
# - os: osx
# On Linux, run with specific JDKs only.
- os: linux
env: CUSTOM_JDK="oraclejdk8"
# On Linux we install latest OpenJDK 1.8 from Ubuntu repositories
- name: Linux x86_64
arch: amd64
- name: Linux aarch64
arch: arm64
cache:
directories:
- $HOME/.m2/repository
before_install:
- lscpu
- echo 'MAVEN_OPTS="$MAVEN_OPTS -Xmx1024m -XX:MaxPermSize=512m -XX:+BytecodeVerificationLocal"' >> ~/.mavenrc
- cat ~/.mavenrc
- if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then export JAVA_HOME=$(/usr/libexec/java_home); fi
- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then jdk_switcher use "$CUSTOM_JDK"; fi
install: |
if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then
sudo apt update
sudo apt install -y openjdk-8-jdk maven
export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-${TRAVIS_CPU_ARCH}/"
export PATH="$JAVA_HOME/bin:/usr/share/maven/bin:$PATH"
fi
before_script:
- java -version
- mvn -version
- ulimit -c unlimited
script:
- travis_retry mvn -B clean apache-rat:check
......
LICENSE
浏览文件 @ 70295453
......@@ -15,7 +15,7 @@
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (properties) the power, direct or indirect, to cause the
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
......
NOTICE
浏览文件 @ 70295453
Apache RocketMQ
Copyright 2016-2020 The Apache Software Foundation
Copyright 2016-2021 The Apache Software Foundation
This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
README.md
浏览文件 @ 70295453
......@@ -6,13 +6,13 @@
[![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html)
[![Average time to resolve an issue](http://isitmaintained.com/badge/resolution/apache/rocketmq.svg)](http://isitmaintained.com/project/apache/rocketmq "Average time to resolve an issue")
[![Percentage of issues still open](http://isitmaintained.com/badge/open/apache/rocketmq.svg)](http://isitmaintained.com/project/apache/rocketmq "Percentage of issues still open")
![Twitter Follow](https://img.shields.io/twitter/follow/ApacheRocketMQ?style=social)
[![Twitter Follow](https://img.shields.io/twitter/follow/ApacheRocketMQ?style=social)](https://twitter.com/intent/follow?screen_name=ApacheRocketMQ)
**[Apache RocketMQ](https://rocketmq.apache.org) is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability.**
It offers a variety of features:
* Messageing patterns including publish/subscribe, request/reply and streaming
* Messaging patterns including publish/subscribe, request/reply and streaming
* Financial grade transactional message
* Built-in fault tolerance and high availability configuration options base on [DLedger](https://github.com/openmessaging/openmessaging-storage-dledger)
* A variety of cross language clients, such as Java, C/C++, Python, Go
......
acl/pom.xml
浏览文件 @ 70295453
......@@ -13,15 +13,11 @@
<parent>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-all</artifactId>
<version>4.7.1-SNAPSHOT</version>
<version>4.9.1-SNAPSHOT</version>
</parent>
<artifactId>rocketmq-acl</artifactId>
<name>rocketmq-acl ${project.version}</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
......@@ -62,11 +58,6 @@
<artifactId>logback-classic</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>commons-validator</groupId>
<artifactId>commons-validator</artifactId>
......
acl/src/main/java/org/apache/rocketmq/acl/common/AclConstants.java
浏览文件 @ 70295453
......@@ -44,6 +44,16 @@ public class AclConstants {
public static final String CONFIG_TIME_STAMP = "timestamp";
public static final String PUB = "PUB";
public static final String SUB = "SUB";
public static final String DENY = "DENY";
public static final String PUB_SUB = "PUB|SUB";
public static final String SUB_PUB = "SUB|PUB";
public static final int ACCESS_KEY_MIN_LENGTH = 6;
public static final int SECRET_KEY_MIN_LENGTH = 6;
......
acl/src/main/java/org/apache/rocketmq/acl/common/AclUtils.java
浏览文件 @ 70295453
......@@ -94,7 +94,7 @@ public class AclUtils {
}
}
public static String v6ipProcess(String netaddress, String[] strArray, int index) {
public static String v6ipProcess(String netaddress) {
int part;
String subAddress;
boolean isAsterisk = isAsterisk(netaddress);
......@@ -120,7 +120,7 @@ public class AclUtils {
}
}
public static String[] getAddreeStrArray(String netaddress, String partialAddress) {
public static String[] getAddresses(String netaddress, String partialAddress) {
String[] parAddStrArray = StringUtils.split(partialAddress.substring(1, partialAddress.length() - 1), ",");
String address = netaddress.substring(0, netaddress.indexOf("{"));
String[] addreeStrArray = new String[parAddStrArray.length];
......
acl/src/main/java/org/apache/rocketmq/acl/common/Permission.java
浏览文件 @ 70295453
......@@ -60,15 +60,14 @@ public class Permission {
return Permission.DENY;
}
switch (permString.trim()) {
case "PUB":
case AclConstants.PUB:
return Permission.PUB;
case "SUB":
case AclConstants.SUB:
return Permission.SUB;
case "PUB|SUB":
case AclConstants.PUB_SUB:
case AclConstants.SUB_PUB:
return Permission.PUB | Permission.SUB;
case "SUB|PUB":
return Permission.PUB | Permission.SUB;
case "DENY":
case AclConstants.DENY:
return Permission.DENY;
default:
return Permission.DENY;
......@@ -90,6 +89,25 @@ public class Permission {
}
}
public static void checkResourcePerms(List<String> resources) {
if (resources == null || resources.isEmpty()) {
return;
}
for (String resource : resources) {
String[] items = StringUtils.split(resource, "=");
if (items.length != 2) {
throw new AclException(String.format("Parse Resource format error for %s.\n" +
"The expected resource format is 'Res=Perm'. For example: topicA=SUB", resource));
}
if (!AclConstants.DENY.equals(items[1].trim()) && Permission.DENY == Permission.parsePermFromString(items[1].trim())) {
throw new AclException(String.format("Parse resource permission error for %s.\n" +
"The expected permissions are 'SUB' or 'PUB' or 'SUB|PUB' or 'PUB|SUB'.", resource));
}
}
}
public static boolean needAdminPerm(Integer code) {
return ADMIN_CODE.contains(code);
}
......
acl/src/main/java/org/apache/rocketmq/acl/plain/PlainPermissionManager.java
浏览文件 @ 70295453
......@@ -50,9 +50,9 @@ public class PlainPermissionManager {
private String fileName = System.getProperty("rocketmq.acl.plain.file", DEFAULT_PLAIN_ACL_FILE);
private Map<String/** AccessKey **/, PlainAccessResource> plainAccessResourceMap = new HashMap<>();
private Map<String/** AccessKey **/, PlainAccessResource> plainAccessResourceMap = new HashMap<>();
private List<RemoteAddressStrategy> globalWhiteRemoteAddressStrategy = new ArrayList<>();
private List<RemoteAddressStrategy> globalWhiteRemoteAddressStrategy = new ArrayList<>();
private RemoteAddressStrategyFactory remoteAddressStrategyFactory = new RemoteAddressStrategyFactory();
......@@ -73,14 +73,14 @@ public class PlainPermissionManager {
JSONObject plainAclConfData = AclUtils.getYamlDataObject(fileHome + File.separator + fileName,
JSONObject.class);
if (plainAclConfData == null || plainAclConfData.isEmpty()) {
throw new AclException(String.format("%s file is not data", fileHome + File.separator + fileName));
throw new AclException(String.format("%s file is not data", fileHome + File.separator + fileName));
}
log.info("Broker plain acl conf data is : ", plainAclConfData.toString());
JSONArray globalWhiteRemoteAddressesList = plainAclConfData.getJSONArray("globalWhiteRemoteAddresses");
if (globalWhiteRemoteAddressesList != null && !globalWhiteRemoteAddressesList.isEmpty()) {
for (int i = 0; i < globalWhiteRemoteAddressesList.size(); i++) {
globalWhiteRemoteAddressStrategy.add(remoteAddressStrategyFactory.
getRemoteAddressStrategy(globalWhiteRemoteAddressesList.getString(i)));
getRemoteAddressStrategy(globalWhiteRemoteAddressesList.getString(i)));
}
}
......@@ -89,7 +89,7 @@ public class PlainPermissionManager {
List<PlainAccessConfig> plainAccessConfigList = accounts.toJavaList(PlainAccessConfig.class);
for (PlainAccessConfig plainAccessConfig : plainAccessConfigList) {
PlainAccessResource plainAccessResource = buildPlainAccessResource(plainAccessConfig);
plainAccessResourceMap.put(plainAccessResource.getAccessKey(),plainAccessResource);
plainAccessResourceMap.put(plainAccessResource.getAccessKey(), plainAccessResource);
}
}
......@@ -128,12 +128,17 @@ public class PlainPermissionManager {
if (plainAccessConfig == null) {
log.error("Parameter value plainAccessConfig is null,Please check your parameter");
return false;
throw new AclException("Parameter value plainAccessConfig is null, Please check your parameter");
}
Permission.checkResourcePerms(plainAccessConfig.getTopicPerms());
Permission.checkResourcePerms(plainAccessConfig.getGroupPerms());
Map<String, Object> aclAccessConfigMap = AclUtils.getYamlDataObject(fileHome + File.separator + fileName,
Map.class);
if (aclAccessConfigMap == null || aclAccessConfigMap.isEmpty()) {
throw new AclException(String.format("the %s file is not found or empty", fileHome + File.separator + fileName));
}
List<Map<String, Object>> accounts = (List<Map<String, Object>>) aclAccessConfigMap.get(AclConstants.CONFIG_ACCOUNTS);
Map<String, Object> updateAccountMap = null;
if (accounts != null) {
......@@ -164,21 +169,21 @@ public class PlainPermissionManager {
return false;
}
private Map<String, Object> createAclAccessConfigMap(Map<String, Object> existedAccoutMap, PlainAccessConfig plainAccessConfig) {
private Map<String, Object> createAclAccessConfigMap(Map<String, Object> existedAccountMap,
PlainAccessConfig plainAccessConfig) {
Map<String, Object> newAccountsMap = null;
if (existedAccoutMap == null) {
if (existedAccountMap == null) {
newAccountsMap = new LinkedHashMap<String, Object>();
} else {
newAccountsMap = existedAccoutMap;
newAccountsMap = existedAccountMap;
}
if (StringUtils.isEmpty(plainAccessConfig.getAccessKey()) ||
plainAccessConfig.getAccessKey().length() <= AclConstants.ACCESS_KEY_MIN_LENGTH) {
throw new AclException(String.format(
"The accessKey=%s cannot be null and length should longer than 6",
plainAccessConfig.getAccessKey()));
"The accessKey=%s cannot be null and length should longer than 6",
plainAccessConfig.getAccessKey()));
}
newAccountsMap.put(AclConstants.CONFIG_ACCESS_KEY, plainAccessConfig.getAccessKey());
......@@ -219,8 +224,10 @@ public class PlainPermissionManager {
}
Map<String, Object> aclAccessConfigMap = AclUtils.getYamlDataObject(fileHome + File.separator + fileName,
Map.class);
Map.class);
if (aclAccessConfigMap == null || aclAccessConfigMap.isEmpty()) {
throw new AclException(String.format("the %s file is not found or empty", fileHome + File.separator + fileName));
}
List<Map<String, Object>> accounts = (List<Map<String, Object>>) aclAccessConfigMap.get("accounts");
if (accounts != null) {
Iterator<Map<String, Object>> itemIterator = accounts.iterator();
......@@ -252,7 +259,9 @@ public class PlainPermissionManager {
Map<String, Object> aclAccessConfigMap = AclUtils.getYamlDataObject(fileHome + File.separator + fileName,
Map.class);
if (aclAccessConfigMap == null || aclAccessConfigMap.isEmpty()) {
throw new AclException(String.format("the %s file is not found or empty", fileHome + File.separator + fileName));
}
List<String> globalWhiteRemoteAddrList = (List<String>) aclAccessConfigMap.get(AclConstants.CONFIG_GLOBAL_WHITE_ADDRS);
if (globalWhiteRemoteAddrList != null) {
......@@ -260,7 +269,7 @@ public class PlainPermissionManager {
globalWhiteRemoteAddrList.addAll(globalWhiteAddrsList);
// Update globalWhiteRemoteAddr element in memeory map firstly
aclAccessConfigMap.put(AclConstants.CONFIG_GLOBAL_WHITE_ADDRS,globalWhiteRemoteAddrList);
aclAccessConfigMap.put(AclConstants.CONFIG_GLOBAL_WHITE_ADDRS, globalWhiteRemoteAddrList);
if (AclUtils.writeDataObject(fileHome + File.separator + fileName, updateAclConfigFileVersion(aclAccessConfigMap))) {
return true;
}
......@@ -276,9 +285,9 @@ public class PlainPermissionManager {
List<PlainAccessConfig> configs = new ArrayList<>();
List<String> whiteAddrs = new ArrayList<>();
JSONObject plainAclConfData = AclUtils.getYamlDataObject(fileHome + File.separator + fileName,
JSONObject.class);
JSONObject.class);
if (plainAclConfData == null || plainAclConfData.isEmpty()) {
throw new AclException(String.format("%s file is not data", fileHome + File.separator + fileName));
throw new AclException(String.format("%s file is not data", fileHome + File.separator + fileName));
}
JSONArray globalWhiteAddrs = plainAclConfData.getJSONArray(AclConstants.CONFIG_GLOBAL_WHITE_ADDRS);
if (globalWhiteAddrs != null && !globalWhiteAddrs.isEmpty()) {
......@@ -360,7 +369,7 @@ public class PlainPermissionManager {
|| plainAccessConfig.getSecretKey().length() <= AclConstants.SECRET_KEY_MIN_LENGTH) {
throw new AclException(String.format(
"The accessKey=%s and secretKey=%s cannot be null and length should longer than 6",
plainAccessConfig.getAccessKey(), plainAccessConfig.getSecretKey()));
plainAccessConfig.getAccessKey(), plainAccessConfig.getSecretKey()));
}
PlainAccessResource plainAccessResource = new PlainAccessResource();
plainAccessResource.setAccessKey(plainAccessConfig.getAccessKey());
......@@ -376,7 +385,7 @@ public class PlainPermissionManager {
Permission.parseResourcePerms(plainAccessResource, true, plainAccessConfig.getTopicPerms());
plainAccessResource.setRemoteAddressStrategy(remoteAddressStrategyFactory.
getRemoteAddressStrategy(plainAccessResource.getWhiteRemoteAddress()));
getRemoteAddressStrategy(plainAccessResource.getWhiteRemoteAddress()));
return plainAccessResource;
}
......
acl/src/main/java/org/apache/rocketmq/acl/plain/RemoteAddressStrategyFactory.java
浏览文件 @ 70295453
......@@ -52,14 +52,18 @@ public class RemoteAddressStrategyFactory {
if (!last.startsWith("{")) {
throw new AclException(String.format("MultipleRemoteAddressStrategy netaddress examine scope Exception netaddress", remoteAddr));
}
return new MultipleRemoteAddressStrategy(AclUtils.getAddreeStrArray(remoteAddr, last));
return new MultipleRemoteAddressStrategy(AclUtils.getAddresses(remoteAddr, last));
} else {
String[] strArray = StringUtils.split(remoteAddr, ".");
String four = strArray[3];
if (!four.startsWith("{")) {
// However a right IP String provided by user,it always can be divided into 4 parts by '.'.
if (strArray.length < 4) {
throw new AclException(String.format("MultipleRemoteAddressStrategy has got a/some wrong format IP(s) ", remoteAddr));
}
String lastStr = strArray[strArray.length - 1];
if (!lastStr.startsWith("{")) {
throw new AclException(String.format("MultipleRemoteAddressStrategy netaddress examine scope Exception netaddress", remoteAddr));
}
return new MultipleRemoteAddressStrategy(AclUtils.getAddreeStrArray(remoteAddr, four));
return new MultipleRemoteAddressStrategy(AclUtils.getAddresses(remoteAddr, lastStr));
}
} else if (AclUtils.isComma(remoteAddr)) {
return new MultipleRemoteAddressStrategy(StringUtils.split(remoteAddr, ","));
......@@ -153,7 +157,7 @@ public class RemoteAddressStrategyFactory {
for (int i = 1; i < strArray.length; i++) {
if (ipv6Analysis(strArray, i)) {
AclUtils.verify(remoteAddr, index - 1);
String preAddress = AclUtils.v6ipProcess(remoteAddr, strArray, index);
String preAddress = AclUtils.v6ipProcess(remoteAddr);
this.index = StringUtils.split(preAddress, ":").length;
this.head = preAddress;
break;
......@@ -189,7 +193,7 @@ public class RemoteAddressStrategyFactory {
throw new AclException(String.format("RangeRemoteAddressStrategy netaddress examine scope Exception start is %s , end is %s", start, end));
}
}
return this.end > 0 ? true : false;
return this.end > 0;
}
private boolean ipv6Analysis(String[] strArray, int index) {
......
acl/src/test/java/org/apache/rocketmq/acl/common/AclUtilsTest.java
浏览文件 @ 70295453
......@@ -32,9 +32,9 @@ import org.junit.Test;
public class AclUtilsTest {
@Test
public void getAddreeStrArray() {
public void getAddresses() {
String address = "1.1.1.{1,2,3,4}";
String[] addressArray = AclUtils.getAddreeStrArray(address, "{1,2,3,4}");
String[] addressArray = AclUtils.getAddresses(address, "{1,2,3,4}");
List<String> newAddressList = new ArrayList<>();
for (String a : addressArray) {
newAddressList.add(a);
......@@ -49,7 +49,7 @@ public class AclUtilsTest {
// IPv6 test
String ipv6Address = "1:ac41:9987::bb22:666:{1,2,3,4}";
String[] ipv6AddressArray = AclUtils.getAddreeStrArray(ipv6Address, "{1,2,3,4}");
String[] ipv6AddressArray = AclUtils.getAddresses(ipv6Address, "{1,2,3,4}");
List<String> newIPv6AddressList = new ArrayList<>();
for (String a : ipv6AddressArray) {
newIPv6AddressList.add(a);
......@@ -181,19 +181,23 @@ public class AclUtilsTest {
public void v6ipProcessTest() {
String remoteAddr = "5::7:6:1-200:*";
String[] strArray = StringUtils.split(remoteAddr, ":");
Assert.assertEquals(AclUtils.v6ipProcess(remoteAddr, strArray, 3), "0005:0000:0000:0000:0007:0006");
Assert.assertEquals(AclUtils.v6ipProcess(remoteAddr), "0005:0000:0000:0000:0007:0006");
// Assert.assertEquals(AclUtils.v6ipProcess(remoteAddr, strArray, 3), "0005:0000:0000:0000:0007:0006");
remoteAddr = "5::7:6:1-200";
strArray = StringUtils.split(remoteAddr, ":");
Assert.assertEquals(AclUtils.v6ipProcess(remoteAddr, strArray, 3), "0005:0000:0000:0000:0000:0007:0006");
Assert.assertEquals(AclUtils.v6ipProcess(remoteAddr), "0005:0000:0000:0000:0000:0007:0006");
// Assert.assertEquals(AclUtils.v6ipProcess(remoteAddr, strArray, 3), "0005:0000:0000:0000:0000:0007:0006");
remoteAddr = "5::7:6:*";
strArray = StringUtils.split(remoteAddr, ":");
Assert.assertEquals(AclUtils.v6ipProcess(remoteAddr, strArray, 3), "0005:0000:0000:0000:0000:0007:0006");
Assert.assertEquals(AclUtils.v6ipProcess(remoteAddr), "0005:0000:0000:0000:0000:0007:0006");
// Assert.assertEquals(AclUtils.v6ipProcess(remoteAddr, strArray, 3), "0005:0000:0000:0000:0000:0007:0006");
remoteAddr = "5:7:6:*";
strArray = StringUtils.split(remoteAddr, ":");
Assert.assertEquals(AclUtils.v6ipProcess(remoteAddr, strArray, 3), "0005:0007:0006");
Assert.assertEquals(AclUtils.v6ipProcess(remoteAddr), "0005:0007:0006");
// Assert.assertEquals(AclUtils.v6ipProcess(remoteAddr, strArray, 3), "0005:0007:0006");
}
@Test
......
acl/src/test/java/org/apache/rocketmq/acl/common/PermissionTest.java
浏览文件 @ 70295453
......@@ -17,6 +17,7 @@
package org.apache.rocketmq.acl.common;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
......@@ -165,4 +166,27 @@ public class PermissionTest {
aclException.setStatus("netaddress examine scope Exception netaddress");
Assert.assertEquals(aclException.getStatus(),"netaddress examine scope Exception netaddress");
}
@Test
public void checkResourcePermsNormalTest() {
Permission.checkResourcePerms(null);
Permission.checkResourcePerms(new ArrayList<>());
Permission.checkResourcePerms(Arrays.asList("topicA=PUB"));
Permission.checkResourcePerms(Arrays.asList("topicA=PUB", "topicB=SUB", "topicC=PUB|SUB"));
}
@Test(expected = AclException.class)
public void checkResourcePermsExceptionTest1() {
Permission.checkResourcePerms(Arrays.asList("topicA"));
}
@Test(expected = AclException.class)
public void checkResourcePermsExceptionTest2() {
Permission.checkResourcePerms(Arrays.asList("topicA="));
}
@Test(expected = AclException.class)
public void checkResourcePermsExceptionTest3() {
Permission.checkResourcePerms(Arrays.asList("topicA=DENY1"));
}
}
acl/src/test/java/org/apache/rocketmq/acl/plain/PlainAccessValidatorTest.java
浏览文件 @ 70295453
......@@ -546,6 +546,26 @@ public class PlainAccessValidatorTest {
Assert.assertEquals(plainAccessValidator.updateAccessConfig(plainAccessConfig), false);
}
@Test(expected = AclException.class)
public void createAndUpdateAccessAclYamlConfigExceptionTest() {
System.setProperty("rocketmq.home.dir", "src/test/resources");
System.setProperty("rocketmq.acl.plain.file", "/conf/plain_acl_update_create.yml");
PlainAccessConfig plainAccessConfig = new PlainAccessConfig();
plainAccessConfig.setAccessKey("RocketMQ33");
plainAccessConfig.setSecretKey("123456789111");
List<String> topicPerms = new ArrayList<String>();
topicPerms.add("topicB=PUB");
plainAccessConfig.setTopicPerms(topicPerms);
List<String> groupPerms = new ArrayList<String>();
groupPerms.add("groupC=DENY1");
plainAccessConfig.setGroupPerms(groupPerms);
PlainAccessValidator plainAccessValidator = new PlainAccessValidator();
// Create element in the acl access yaml config file
plainAccessValidator.updateAccessConfig(plainAccessConfig);
}
@Test
public void updateGlobalWhiteAddrsNormalTest() {
System.setProperty("rocketmq.home.dir", "src/test/resources");
......
acl/src/test/java/org/apache/rocketmq/acl/plain/RemoteAddressStrategyTest.java
浏览文件 @ 70295453
......@@ -198,6 +198,16 @@ public class RemoteAddressStrategyTest {
remoteAddressStrategyFactory.getRemoteAddressStrategy(plainAccessResource);
plainAccessResource.setWhiteRemoteAddress("::1,2,3}");
remoteAddressStrategyFactory.getRemoteAddressStrategy(plainAccessResource);
plainAccessResource.setWhiteRemoteAddress("192.168.1.{1}");
remoteAddressStrategyFactory.getRemoteAddressStrategy(plainAccessResource);
plainAccessResource.setWhiteRemoteAddress("192.168.1.{1,2}");
remoteAddressStrategyFactory.getRemoteAddressStrategy(plainAccessResource);
plainAccessResource.setWhiteRemoteAddress("192.168.{1}");
remoteAddressStrategyFactory.getRemoteAddressStrategy(plainAccessResource);
plainAccessResource.setWhiteRemoteAddress("{192.168.1}");
remoteAddressStrategyFactory.getRemoteAddressStrategy(plainAccessResource);
plainAccessResource.setWhiteRemoteAddress("{192.168.1.1}");
remoteAddressStrategyFactory.getRemoteAddressStrategy(plainAccessResource);
}
private void multipleNetaddressStrategyTest(RemoteAddressStrategy remoteAddressStrategy) {
......
broker/pom.xml
浏览文件 @ 70295453
......@@ -13,7 +13,7 @@
<parent>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-all</artifactId>
<version>4.7.1-SNAPSHOT</version>
<version>4.9.1-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
......@@ -54,10 +54,6 @@
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
......
broker/src/main/java/org/apache/rocketmq/broker/client/ProducerManager.java
浏览文件 @ 70295453
......@@ -33,7 +33,7 @@ import org.apache.rocketmq.remoting.common.RemotingUtil;
public class ProducerManager {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
private static final long CHANNEL_EXPIRED_TIMEOUT = 1000 * 120;
private static final int GET_AVALIABLE_CHANNEL_RETRY_COUNT = 3;
private static final int GET_AVAILABLE_CHANNEL_RETRY_COUNT = 3;
private final ConcurrentHashMap<String /* group name */, ConcurrentHashMap<Channel, ClientChannelInfo>> groupChannelTable =
new ConcurrentHashMap<>();
private final ConcurrentHashMap<String, Channel> clientChannelTable = new ConcurrentHashMap<>();
......@@ -131,16 +131,14 @@ public class ProducerManager {
}
}
public Channel getAvaliableChannel(String groupId) {
public Channel getAvailableChannel(String groupId) {
if (groupId == null) {
return null;
}
List<Channel> channelList = new ArrayList<Channel>();
List<Channel> channelList;
ConcurrentHashMap<Channel, ClientChannelInfo> channelClientChannelInfoHashMap = groupChannelTable.get(groupId);
if (channelClientChannelInfoHashMap != null) {
for (Channel channel : channelClientChannelInfoHashMap.keySet()) {
channelList.add(channel);
}
channelList = new ArrayList<>(channelClientChannelInfoHashMap.keySet());
} else {
log.warn("Check transaction failed, channel table is empty. groupId={}", groupId);
return null;
......@@ -158,7 +156,7 @@ public class ProducerManager {
Channel channel = channelList.get(index);
int count = 0;
boolean isOk = channel.isActive() && channel.isWritable();
while (count++ < GET_AVALIABLE_CHANNEL_RETRY_COUNT) {
while (count++ < GET_AVAILABLE_CHANNEL_RETRY_COUNT) {
if (isOk) {
return channel;
}
......
broker/src/main/java/org/apache/rocketmq/broker/mqtrace/SendMessageHook.java
浏览文件 @ 70295453
......@@ -17,9 +17,9 @@
package org.apache.rocketmq.broker.mqtrace;
public interface SendMessageHook {
public String hookName();
String hookName();
public void sendMessageBefore(final SendMessageContext context);
void sendMessageBefore(final SendMessageContext context);
public void sendMessageAfter(final SendMessageContext context);
void sendMessageAfter(final SendMessageContext context);
}
broker/src/main/java/org/apache/rocketmq/broker/offset/ConsumerOffsetManager.java
浏览文件 @ 70295453
......@@ -232,4 +232,20 @@ public class ConsumerOffsetManager extends ConfigManager {
}
}
public void removeOffset(final String group) {
Iterator<Entry<String, ConcurrentMap<Integer, Long>>> it = this.offsetTable.entrySet().iterator();
while (it.hasNext()) {
Entry<String, ConcurrentMap<Integer, Long>> next = it.next();
String topicAtGroup = next.getKey();
if (topicAtGroup.contains(group)) {
String[] arrays = topicAtGroup.split(TOPIC_GROUP_SEPARATOR);
if (arrays.length == 2 && group.equals(arrays[1])) {
it.remove();
log.warn("clean group offset {}", topicAtGroup);
}
}
}
}
}
broker/src/main/java/org/apache/rocketmq/broker/out/BrokerOuterAPI.java
浏览文件 @ 70295453
......@@ -16,7 +16,6 @@
*/
package org.apache.rocketmq.broker.out;
import com.google.common.collect.Lists;
import java.io.UnsupportedEncodingException;
import java.util.ArrayList;
import java.util.List;
......@@ -123,7 +122,7 @@ public class BrokerOuterAPI {
final int timeoutMills,
final boolean compressed) {
final List<RegisterBrokerResult> registerBrokerResultList = Lists.newArrayList();
final List<RegisterBrokerResult> registerBrokerResultList = new CopyOnWriteArrayList<>();
List<String> nameServerAddressList = this.remotingClient.getNameServerAddressList();
if (nameServerAddressList != null && nameServerAddressList.size() > 0) {
......@@ -147,7 +146,7 @@ public class BrokerOuterAPI {
@Override
public void run() {
try {
RegisterBrokerResult result = registerBroker(namesrvAddr,oneway, timeoutMills,requestHeader,body);
RegisterBrokerResult result = registerBroker(namesrvAddr, oneway, timeoutMills, requestHeader, body);
if (result != null) {
registerBrokerResultList.add(result);
}
......@@ -209,7 +208,7 @@ public class BrokerOuterAPI {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), requestHeader == null ? null : requestHeader.getBrokerAddr());
}
public void unregisterBrokerAll(
......@@ -255,7 +254,7 @@ public class BrokerOuterAPI {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), brokerAddr);
}
public List<Boolean> needRegister(
......@@ -338,7 +337,7 @@ public class BrokerOuterAPI {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public ConsumerOffsetSerializeWrapper getAllConsumerOffset(
......@@ -355,7 +354,7 @@ public class BrokerOuterAPI {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public String getAllDelayOffset(
......@@ -372,7 +371,7 @@ public class BrokerOuterAPI {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public SubscriptionGroupWrapper getAllSubscriptionGroupConfig(
......@@ -389,7 +388,7 @@ public class BrokerOuterAPI {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public void registerRPCHook(RPCHook rpcHook) {
......
broker/src/main/java/org/apache/rocketmq/broker/processor/AbstractSendMessageProcessor.java
浏览文件 @ 70295453
......@@ -25,7 +25,7 @@ import java.util.Random;
import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.broker.mqtrace.SendMessageContext;
import org.apache.rocketmq.broker.mqtrace.SendMessageHook;
import org.apache.rocketmq.broker.topic.TopicValidator;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.TopicConfig;
import org.apache.rocketmq.common.TopicFilterType;
......@@ -176,6 +176,9 @@ public abstract class AbstractSendMessageProcessor extends AsyncNettyRequestProc
if (!TopicValidator.validateTopic(requestHeader.getTopic(), response)) {
return response;
}
if (TopicValidator.isNotAllowedSendTopic(requestHeader.getTopic(), response)) {
return response;
}
TopicConfig topicConfig =
this.brokerController.getTopicConfigManager().selectTopicConfig(requestHeader.getTopic());
......
broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
浏览文件 @ 70295453
......@@ -19,17 +19,6 @@ package org.apache.rocketmq.broker.processor;
import com.alibaba.fastjson.JSON;
import io.netty.channel.Channel;
import io.netty.channel.ChannelHandlerContext;
import java.io.UnsupportedEncodingException;
import java.net.UnknownHostException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.Set;
import java.util.concurrent.ConcurrentMap;
import org.apache.rocketmq.acl.AccessValidator;
import org.apache.rocketmq.acl.plain.PlainAccessValidator;
import org.apache.rocketmq.broker.BrokerController;
......@@ -37,8 +26,8 @@ import org.apache.rocketmq.broker.client.ClientChannelInfo;
import org.apache.rocketmq.broker.client.ConsumerGroupInfo;
import org.apache.rocketmq.broker.filter.ConsumerFilterData;
import org.apache.rocketmq.broker.filter.ExpressionMessageFilter;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.broker.transaction.queue.TransactionalMessageUtil;
import org.apache.rocketmq.broker.topic.TopicValidator;
import org.apache.rocketmq.common.AclConfig;
import org.apache.rocketmq.common.MQVersion;
import org.apache.rocketmq.common.MixAll;
......@@ -127,6 +116,7 @@ import org.apache.rocketmq.remoting.netty.NettyRequestProcessor;
import org.apache.rocketmq.remoting.protocol.LanguageCode;
import org.apache.rocketmq.remoting.protocol.RemotingCommand;
import org.apache.rocketmq.remoting.protocol.RemotingSerializable;
import org.apache.rocketmq.remoting.protocol.RemotingSysResponseCode;
import org.apache.rocketmq.store.ConsumeQueue;
import org.apache.rocketmq.store.ConsumeQueueExt;
import org.apache.rocketmq.store.DefaultMessageStore;
......@@ -137,6 +127,18 @@ import org.apache.rocketmq.store.PutMessageResult;
import org.apache.rocketmq.store.PutMessageStatus;
import org.apache.rocketmq.store.SelectMappedBufferResult;
import java.io.UnsupportedEncodingException;
import java.net.UnknownHostException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.Set;
import java.util.concurrent.ConcurrentMap;
public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements NettyRequestProcessor {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
private final BrokerController brokerController;
......@@ -234,10 +236,8 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
case RequestCode.GET_BROKER_CLUSTER_ACL_CONFIG:
return getBrokerClusterAclConfig(ctx, request);
default:
break;
return getUnknownCmdResponse(ctx, request);
}
return null;
}
@Override
......@@ -252,29 +252,16 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
(CreateTopicRequestHeader) request.decodeCommandCustomHeader(CreateTopicRequestHeader.class);
log.info("updateAndCreateTopic called by {}", RemotingHelper.parseChannelRemoteAddr(ctx.channel()));
if (requestHeader.getTopic().equals(this.brokerController.getBrokerConfig().getBrokerClusterName())) {
String errorMsg = "the topic[" + requestHeader.getTopic() + "] is conflict with system reserved words.";
log.warn(errorMsg);
response.setCode(ResponseCode.SYSTEM_ERROR);
response.setRemark(errorMsg);
return response;
}
String topic = requestHeader.getTopic();
if (!TopicValidator.validateTopic(requestHeader.getTopic(), response)) {
if (!TopicValidator.validateTopic(topic, response)) {
return response;
}
try {
response.setCode(ResponseCode.SUCCESS);
response.setOpaque(request.getOpaque());
response.markResponseType();
response.setRemark(null);
ctx.writeAndFlush(response);
} catch (Exception e) {
log.error("Failed to produce a proper response", e);
if (TopicValidator.isSystemTopic(topic, response)) {
return response;
}
TopicConfig topicConfig = new TopicConfig(requestHeader.getTopic());
TopicConfig topicConfig = new TopicConfig(topic);
topicConfig.setReadQueueNums(requestHeader.getReadQueueNums());
topicConfig.setWriteQueueNums(requestHeader.getWriteQueueNums());
topicConfig.setTopicFilterType(requestHeader.getTopicFilterTypeEnum());
......@@ -285,7 +272,8 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
this.brokerController.registerIncrementBrokerData(topicConfig, this.brokerController.getTopicConfigManager().getDataVersion());
return null;
response.setCode(ResponseCode.SUCCESS);
return response;
}
private synchronized RemotingCommand deleteTopic(ChannelHandlerContext ctx,
......@@ -296,7 +284,15 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
log.info("deleteTopic called by {}", RemotingHelper.parseChannelRemoteAddr(ctx.channel()));
this.brokerController.getTopicConfigManager().deleteTopicConfig(requestHeader.getTopic());
String topic = requestHeader.getTopic();
if (!TopicValidator.validateTopic(topic, response)) {
return response;
}
if (TopicValidator.isSystemTopic(topic, response)) {
return response;
}
this.brokerController.getTopicConfigManager().deleteTopicConfig(topic);
this.brokerController.getMessageStore()
.cleanUnusedTopic(this.brokerController.getTopicConfigManager().getTopicConfigTable().keySet());
if (this.brokerController.getBrokerConfig().isAutoDeleteUnusedStats()) {
......@@ -430,7 +426,7 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
responseHeader.setBrokerAddr(this.brokerController.getBrokerAddr());
responseHeader.setBrokerName(this.brokerController.getBrokerConfig().getBrokerName());
responseHeader.setClusterName(this.brokerController.getBrokerConfig().getBrokerClusterName());
response.setCode(ResponseCode.SUCCESS);
response.setRemark(null);
return response;
......@@ -462,6 +458,13 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
return null;
}
private RemotingCommand getUnknownCmdResponse(ChannelHandlerContext ctx, RemotingCommand request) {
String error = " request type " + request.getCode() + " not supported";
final RemotingCommand response =
RemotingCommand.createResponseCommand(RemotingSysResponseCode.REQUEST_CODE_NOT_SUPPORTED, error);
return response;
}
private RemotingCommand getAllTopicConfig(ChannelHandlerContext ctx, RemotingCommand request) {
final RemotingCommand response = RemotingCommand.createResponseCommand(GetAllTopicConfigResponseHeader.class);
// final GetAllTopicConfigResponseHeader responseHeader =
......@@ -717,6 +720,10 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
this.brokerController.getSubscriptionGroupManager().deleteSubscriptionGroupConfig(requestHeader.getGroupName());
if (requestHeader.isRemoveOffset()) {
this.brokerController.getConsumerOffsetManager().removeOffset(requestHeader.getGroupName());
}
if (this.brokerController.getBrokerConfig().isAutoDeleteUnusedStats()) {
this.brokerController.getBrokerStatsManager().onGroupDeleted(requestHeader.getGroupName());
}
......@@ -1118,7 +1125,7 @@ public class AdminBrokerProcessor extends AsyncNettyRequestProcessor implements
throws RemotingCommandException {
final RemotingCommand response = RemotingCommand.createResponseCommand(null);
Set<String> topics = this.brokerController.getTopicConfigManager().getSystemTopic();
Set<String> topics = TopicValidator.getSystemTopicSet();
TopicList topicList = new TopicList();
topicList.setTopicList(topics);
response.setBody(topicList.encode());
......
broker/src/main/java/org/apache/rocketmq/broker/processor/PullMessageProcessor.java
浏览文件 @ 70295453
......@@ -52,6 +52,7 @@ import org.apache.rocketmq.common.protocol.topic.OffsetMovedEvent;
import org.apache.rocketmq.common.subscription.SubscriptionGroupConfig;
import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.apache.rocketmq.common.sysflag.PullSysFlag;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.remoting.common.RemotingHelper;
......@@ -523,7 +524,7 @@ public class PullMessageProcessor extends AsyncNettyRequestProcessor implements
private void generateOffsetMovedEvent(final OffsetMovedEvent event) {
try {
MessageExtBrokerInner msgInner = new MessageExtBrokerInner();
msgInner.setTopic(MixAll.OFFSET_MOVED_EVENT);
msgInner.setTopic(TopicValidator.RMQ_SYS_OFFSET_MOVED_EVENT);
msgInner.setTags(event.getConsumerGroup());
msgInner.setDelayTimeLevel(0);
msgInner.setKeys(event.getConsumerGroup());
......@@ -583,7 +584,7 @@ public class PullMessageProcessor extends AsyncNettyRequestProcessor implements
this.brokerController.getPullMessageExecutor().submit(new RequestTask(run, channel, request));
}
public void registerConsumeMessageHook(List<ConsumeMessageHook> sendMessageHookList) {
this.consumeMessageHookList = sendMessageHookList;
public void registerConsumeMessageHook(List<ConsumeMessageHook> consumeMessageHookList) {
this.consumeMessageHookList = consumeMessageHookList;
}
}
broker/src/main/java/org/apache/rocketmq/broker/processor/ReplyMessageProcessor.java
浏览文件 @ 70295453
......@@ -251,28 +251,29 @@ public class ReplyMessageProcessor extends AbstractSendMessageProcessor implemen
// Failed
case CREATE_MAPEDFILE_FAILED:
log.info("create mapped file failed, server is busy or broken.");
log.warn("create mapped file failed, server is busy or broken.");
break;
case MESSAGE_ILLEGAL:
log.info(
log.warn(
"the message is illegal, maybe msg properties length limit 32k.");
break;
case PROPERTIES_SIZE_EXCEEDED:
log.info(
log.warn(
"the message is illegal, maybe msg body or properties length not matched. msg body length limit 128k.");
break;
case SERVICE_NOT_AVAILABLE:
log.info(
"service not available now, maybe disk full, maybe your broker machine memory too small.");
log.warn(
"service not available now. It may be caused by one of the following reasons: " +
"the broker's disk is full, messages are put to the slave, message store has been shut down, etc.");
break;
case OS_PAGECACHE_BUSY:
log.info("[PC_SYNCHRONIZED]broker busy, start flow control for a while");
log.warn("[PC_SYNCHRONIZED]broker busy, start flow control for a while");
break;
case UNKNOWN_ERROR:
log.info("UNKNOWN_ERROR");
log.warn("UNKNOWN_ERROR");
break;
default:
log.info("UNKNOWN_ERROR DEFAULT");
log.warn("UNKNOWN_ERROR DEFAULT");
break;
}
......
broker/src/main/java/org/apache/rocketmq/broker/processor/SendMessageProcessor.java
浏览文件 @ 70295453
......@@ -218,6 +218,8 @@ public class SendMessageProcessor extends AbstractSendMessageProcessor implement
String originMsgId = MessageAccessor.getOriginMessageId(msgExt);
MessageAccessor.setOriginMessageId(msgInner, UtilAll.isBlank(originMsgId) ? msgExt.getMsgId() : originMsgId);
msgInner.setPropertiesString(MessageDecoder.messageProperties2String(msgExt.getProperties()));
CompletableFuture<PutMessageResult> putMessageResult = this.brokerController.getMessageStore().asyncPutMessage(msgInner);
return putMessageResult.thenApply((r) -> {
if (r != null) {
......@@ -480,7 +482,8 @@ public class SendMessageProcessor extends AbstractSendMessageProcessor implement
case SERVICE_NOT_AVAILABLE:
response.setCode(ResponseCode.SERVICE_NOT_AVAILABLE);
response.setRemark(
"service not available now, maybe disk full, " + diskUtil() + ", maybe your broker machine memory too small.");
"service not available now. It may be caused by one of the following reasons: " +
"the broker's disk is full [" + diskUtil() + "], messages are put to the slave, message store has been shut down, etc.");
break;
case OS_PAGECACHE_BUSY:
response.setCode(ResponseCode.SYSTEM_ERROR);
......
broker/src/main/java/org/apache/rocketmq/broker/topic/TopicConfigManager.java
浏览文件 @ 70295453
......@@ -16,7 +16,6 @@
*/
package org.apache.rocketmq.broker.topic;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Map;
import java.util.Map.Entry;
......@@ -37,18 +36,20 @@ import org.apache.rocketmq.common.constant.PermName;
import org.apache.rocketmq.common.protocol.body.KVTable;
import org.apache.rocketmq.common.protocol.body.TopicConfigSerializeWrapper;
import org.apache.rocketmq.common.sysflag.TopicSysFlag;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
public class TopicConfigManager extends ConfigManager {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
private static final long LOCK_TIMEOUT_MILLIS = 3000;
private transient final Lock lockTopicConfigTable = new ReentrantLock();
private static final int SCHEDULE_TOPIC_QUEUE_NUM = 18;
private transient final Lock topicConfigTableLock = new ReentrantLock();
private final ConcurrentMap<String, TopicConfig> topicConfigTable =
new ConcurrentHashMap<String, TopicConfig>(1024);
private final DataVersion dataVersion = new DataVersion();
private final Set<String> systemTopicList = new HashSet<String>();
private transient BrokerController brokerController;
public TopicConfigManager() {
......@@ -57,20 +58,18 @@ public class TopicConfigManager extends ConfigManager {
public TopicConfigManager(BrokerController brokerController) {
this.brokerController = brokerController;
{
// MixAll.SELF_TEST_TOPIC
String topic = MixAll.SELF_TEST_TOPIC;
String topic = TopicValidator.RMQ_SYS_SELF_TEST_TOPIC;
TopicConfig topicConfig = new TopicConfig(topic);
this.systemTopicList.add(topic);
TopicValidator.addSystemTopic(topic);
topicConfig.setReadQueueNums(1);
topicConfig.setWriteQueueNums(1);
this.topicConfigTable.put(topicConfig.getTopicName(), topicConfig);
}
{
// MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC
if (this.brokerController.getBrokerConfig().isAutoCreateTopicEnable()) {
String topic = MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC;
String topic = TopicValidator.AUTO_CREATE_TOPIC_KEY_TOPIC;
TopicConfig topicConfig = new TopicConfig(topic);
this.systemTopicList.add(topic);
TopicValidator.addSystemTopic(topic);
topicConfig.setReadQueueNums(this.brokerController.getBrokerConfig()
.getDefaultTopicQueueNums());
topicConfig.setWriteQueueNums(this.brokerController.getBrokerConfig()
......@@ -81,10 +80,9 @@ public class TopicConfigManager extends ConfigManager {
}
}
{
// MixAll.BENCHMARK_TOPIC
String topic = MixAll.BENCHMARK_TOPIC;
String topic = TopicValidator.RMQ_SYS_BENCHMARK_TOPIC;
TopicConfig topicConfig = new TopicConfig(topic);
this.systemTopicList.add(topic);
TopicValidator.addSystemTopic(topic);
topicConfig.setReadQueueNums(1024);
topicConfig.setWriteQueueNums(1024);
this.topicConfigTable.put(topicConfig.getTopicName(), topicConfig);
......@@ -93,7 +91,7 @@ public class TopicConfigManager extends ConfigManager {
String topic = this.brokerController.getBrokerConfig().getBrokerClusterName();
TopicConfig topicConfig = new TopicConfig(topic);
this.systemTopicList.add(topic);
TopicValidator.addSystemTopic(topic);
int perm = PermName.PERM_INHERIT;
if (this.brokerController.getBrokerConfig().isClusterTopicEnable()) {
perm |= PermName.PERM_READ | PermName.PERM_WRITE;
......@@ -105,7 +103,7 @@ public class TopicConfigManager extends ConfigManager {
String topic = this.brokerController.getBrokerConfig().getBrokerName();
TopicConfig topicConfig = new TopicConfig(topic);
this.systemTopicList.add(topic);
TopicValidator.addSystemTopic(topic);
int perm = PermName.PERM_INHERIT;
if (this.brokerController.getBrokerConfig().isBrokerTopicEnable()) {
perm |= PermName.PERM_READ | PermName.PERM_WRITE;
......@@ -116,19 +114,26 @@ public class TopicConfigManager extends ConfigManager {
this.topicConfigTable.put(topicConfig.getTopicName(), topicConfig);
}
{
// MixAll.OFFSET_MOVED_EVENT
String topic = MixAll.OFFSET_MOVED_EVENT;
String topic = TopicValidator.RMQ_SYS_OFFSET_MOVED_EVENT;
TopicConfig topicConfig = new TopicConfig(topic);
this.systemTopicList.add(topic);
TopicValidator.addSystemTopic(topic);
topicConfig.setReadQueueNums(1);
topicConfig.setWriteQueueNums(1);
this.topicConfigTable.put(topicConfig.getTopicName(), topicConfig);
}
{
String topic = TopicValidator.RMQ_SYS_SCHEDULE_TOPIC;
TopicConfig topicConfig = new TopicConfig(topic);
TopicValidator.addSystemTopic(topic);
topicConfig.setReadQueueNums(SCHEDULE_TOPIC_QUEUE_NUM);
topicConfig.setWriteQueueNums(SCHEDULE_TOPIC_QUEUE_NUM);
this.topicConfigTable.put(topicConfig.getTopicName(), topicConfig);
}
{
if (this.brokerController.getBrokerConfig().isTraceTopicEnable()) {
String topic = this.brokerController.getBrokerConfig().getMsgTraceTopicName();
TopicConfig topicConfig = new TopicConfig(topic);
this.systemTopicList.add(topic);
TopicValidator.addSystemTopic(topic);
topicConfig.setReadQueueNums(1);
topicConfig.setWriteQueueNums(1);
this.topicConfigTable.put(topicConfig.getTopicName(), topicConfig);
......@@ -137,21 +142,13 @@ public class TopicConfigManager extends ConfigManager {
{
String topic = this.brokerController.getBrokerConfig().getBrokerClusterName() + "_" + MixAll.REPLY_TOPIC_POSTFIX;
TopicConfig topicConfig = new TopicConfig(topic);
this.systemTopicList.add(topic);
TopicValidator.addSystemTopic(topic);
topicConfig.setReadQueueNums(1);
topicConfig.setWriteQueueNums(1);
this.topicConfigTable.put(topicConfig.getTopicName(), topicConfig);
}
}
public boolean isSystemTopic(final String topic) {
return this.systemTopicList.contains(topic);
}
public Set<String> getSystemTopic() {
return this.systemTopicList;
}
public TopicConfig selectTopicConfig(final String topic) {
return this.topicConfigTable.get(topic);
}
......@@ -162,7 +159,7 @@ public class TopicConfigManager extends ConfigManager {
boolean createNew = false;
try {
if (this.lockTopicConfigTable.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
if (this.topicConfigTableLock.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
try {
topicConfig = this.topicConfigTable.get(topic);
if (topicConfig != null)
......@@ -170,7 +167,7 @@ public class TopicConfigManager extends ConfigManager {
TopicConfig defaultTopicConfig = this.topicConfigTable.get(defaultTopic);
if (defaultTopicConfig != null) {
if (defaultTopic.equals(MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC)) {
if (defaultTopic.equals(TopicValidator.AUTO_CREATE_TOPIC_KEY_TOPIC)) {
if (!this.brokerController.getBrokerConfig().isAutoCreateTopicEnable()) {
defaultTopicConfig.setPerm(PermName.PERM_READ | PermName.PERM_WRITE);
}
......@@ -179,9 +176,7 @@ public class TopicConfigManager extends ConfigManager {
if (PermName.isInherited(defaultTopicConfig.getPerm())) {
topicConfig = new TopicConfig(topic);
int queueNums =
clientDefaultTopicQueueNums > defaultTopicConfig.getWriteQueueNums() ? defaultTopicConfig
.getWriteQueueNums() : clientDefaultTopicQueueNums;
int queueNums = Math.min(clientDefaultTopicQueueNums, defaultTopicConfig.getWriteQueueNums());
if (queueNums < 0) {
queueNums = 0;
......@@ -216,7 +211,7 @@ public class TopicConfigManager extends ConfigManager {
this.persist();
}
} finally {
this.lockTopicConfigTable.unlock();
this.topicConfigTableLock.unlock();
}
}
} catch (InterruptedException e) {
......@@ -242,7 +237,7 @@ public class TopicConfigManager extends ConfigManager {
boolean createNew = false;
try {
if (this.lockTopicConfigTable.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
if (this.topicConfigTableLock.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
try {
topicConfig = this.topicConfigTable.get(topic);
if (topicConfig != null)
......@@ -260,7 +255,7 @@ public class TopicConfigManager extends ConfigManager {
this.dataVersion.nextVersion();
this.persist();
} finally {
this.lockTopicConfigTable.unlock();
this.topicConfigTableLock.unlock();
}
}
} catch (InterruptedException e) {
......@@ -275,32 +270,32 @@ public class TopicConfigManager extends ConfigManager {
}
public TopicConfig createTopicOfTranCheckMaxTime(final int clientDefaultTopicQueueNums, final int perm) {
TopicConfig topicConfig = this.topicConfigTable.get(MixAll.TRANS_CHECK_MAX_TIME_TOPIC);
TopicConfig topicConfig = this.topicConfigTable.get(TopicValidator.RMQ_SYS_TRANS_CHECK_MAX_TIME_TOPIC);
if (topicConfig != null)
return topicConfig;
boolean createNew = false;
try {
if (this.lockTopicConfigTable.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
if (this.topicConfigTableLock.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
try {
topicConfig = this.topicConfigTable.get(MixAll.TRANS_CHECK_MAX_TIME_TOPIC);
topicConfig = this.topicConfigTable.get(TopicValidator.RMQ_SYS_TRANS_CHECK_MAX_TIME_TOPIC);
if (topicConfig != null)
return topicConfig;
topicConfig = new TopicConfig(MixAll.TRANS_CHECK_MAX_TIME_TOPIC);
topicConfig = new TopicConfig(TopicValidator.RMQ_SYS_TRANS_CHECK_MAX_TIME_TOPIC);
topicConfig.setReadQueueNums(clientDefaultTopicQueueNums);
topicConfig.setWriteQueueNums(clientDefaultTopicQueueNums);
topicConfig.setPerm(perm);
topicConfig.setTopicSysFlag(0);
log.info("create new topic {}", topicConfig);
this.topicConfigTable.put(MixAll.TRANS_CHECK_MAX_TIME_TOPIC, topicConfig);
this.topicConfigTable.put(TopicValidator.RMQ_SYS_TRANS_CHECK_MAX_TIME_TOPIC, topicConfig);
createNew = true;
this.dataVersion.nextVersion();
this.persist();
} finally {
this.lockTopicConfigTable.unlock();
this.topicConfigTableLock.unlock();
}
}
} catch (InterruptedException e) {
......
broker/src/main/java/org/apache/rocketmq/broker/transaction/AbstractTransactionalMessageCheckListener.java
浏览文件 @ 70295453
......@@ -69,7 +69,7 @@ public abstract class AbstractTransactionalMessageCheckListener {
msgExt.setQueueId(Integer.parseInt(msgExt.getUserProperty(MessageConst.PROPERTY_REAL_QUEUE_ID)));
msgExt.setStoreSize(0);
String groupId = msgExt.getProperty(MessageConst.PROPERTY_PRODUCER_GROUP);
Channel channel = brokerController.getProducerManager().getAvaliableChannel(groupId);
Channel channel = brokerController.getProducerManager().getAvailableChannel(groupId);
if (channel != null) {
brokerController.getBroker2Client().checkProducerTransactionState(groupId, channel, checkTransactionStateRequestHeader, msgExt);
} else {
......
broker/src/main/java/org/apache/rocketmq/broker/transaction/queue/TransactionalMessageServiceImpl.java
浏览文件 @ 70295453
......@@ -21,13 +21,13 @@ import org.apache.rocketmq.broker.transaction.OperationResult;
import org.apache.rocketmq.broker.transaction.TransactionalMessageService;
import org.apache.rocketmq.client.consumer.PullResult;
import org.apache.rocketmq.client.consumer.PullStatus;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.constant.LoggerName;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.protocol.ResponseCode;
import org.apache.rocketmq.common.protocol.header.EndTransactionRequestHeader;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.logging.InternalLoggerFactory;
import org.apache.rocketmq.store.MessageExtBrokerInner;
......@@ -127,7 +127,7 @@ public class TransactionalMessageServiceImpl implements TransactionalMessageServ
public void check(long transactionTimeout, int transactionCheckMax,
AbstractTransactionalMessageCheckListener listener) {
try {
String topic = MixAll.RMQ_SYS_TRANS_HALF_TOPIC;
String topic = TopicValidator.RMQ_SYS_TRANS_HALF_TOPIC;
Set<MessageQueue> msgQueues = transactionalMessageBridge.fetchMessageQueues(topic);
if (msgQueues == null || msgQueues.size() == 0) {
log.warn("The queue of topic is empty :" + topic);
......@@ -164,7 +164,7 @@ public class TransactionalMessageServiceImpl implements TransactionalMessageServ
break;
}
if (removeMap.containsKey(i)) {
log.info("Half offset {} has been committed/rolled back", i);
log.debug("Half offset {} has been committed/rolled back", i);
Long removedOpOffset = removeMap.remove(i);
doneOpOffset.add(removedOpOffset);
} else {
......
broker/src/main/java/org/apache/rocketmq/broker/transaction/queue/TransactionalMessageUtil.java
浏览文件 @ 70295453
......@@ -17,6 +17,7 @@
package org.apache.rocketmq.broker.transaction.queue;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.topic.TopicValidator;
import java.nio.charset.Charset;
......@@ -25,11 +26,11 @@ public class TransactionalMessageUtil {
public static Charset charset = Charset.forName("utf-8");
public static String buildOpTopic() {
return MixAll.RMQ_SYS_TRANS_OP_HALF_TOPIC;
return TopicValidator.RMQ_SYS_TRANS_OP_HALF_TOPIC;
}
public static String buildHalfTopic() {
return MixAll.RMQ_SYS_TRANS_HALF_TOPIC;
return TopicValidator.RMQ_SYS_TRANS_HALF_TOPIC;
}
public static String buildConsumerGroup() {
......
broker/src/test/java/org/apache/rocketmq/broker/BrokerOuterAPITest.java
浏览文件 @ 70295453
......@@ -124,7 +124,7 @@ public class BrokerOuterAPITest {
boolean success = Iterables.any(booleanList,
new Predicate<Boolean>() {
public boolean apply(Boolean input) {
return input ? true : false;
return input;
}
});
......
broker/src/test/java/org/apache/rocketmq/broker/client/ProducerManagerTest.java
浏览文件 @ 70295453
......@@ -110,20 +110,20 @@ public class ProducerManagerTest {
}
@Test
public void testGetAvaliableChannel() {
public void testGetAvailableChannel() {
producerManager.registerProducer(group, clientInfo);
when(channel.isActive()).thenReturn(true);
when(channel.isWritable()).thenReturn(true);
Channel c = producerManager.getAvaliableChannel(group);
Channel c = producerManager.getAvailableChannel(group);
assertThat(c).isSameAs(channel);
when(channel.isWritable()).thenReturn(false);
c = producerManager.getAvaliableChannel(group);
c = producerManager.getAvailableChannel(group);
assertThat(c).isSameAs(channel);
when(channel.isActive()).thenReturn(false);
c = producerManager.getAvaliableChannel(group);
c = producerManager.getAvailableChannel(group);
assertThat(c).isNull();
}
......
broker/src/test/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessorTest.java
浏览文件 @ 70295453
......@@ -16,17 +16,22 @@
*/
package org.apache.rocketmq.broker.processor;
import com.google.common.collect.Sets;
import io.netty.channel.ChannelHandlerContext;
import java.net.UnknownHostException;
import java.nio.ByteBuffer;
import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.common.BrokerConfig;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.TopicFilterType;
import org.apache.rocketmq.common.constant.PermName;
import org.apache.rocketmq.common.message.MessageAccessor;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.protocol.RequestCode;
import org.apache.rocketmq.common.protocol.ResponseCode;
import org.apache.rocketmq.common.protocol.header.CreateTopicRequestHeader;
import org.apache.rocketmq.common.protocol.header.DeleteTopicRequestHeader;
import org.apache.rocketmq.common.protocol.header.ResumeCheckHalfMessageRequestHeader;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
import org.apache.rocketmq.remoting.netty.NettyClientConfig;
import org.apache.rocketmq.remoting.netty.NettyServerConfig;
......@@ -47,6 +52,10 @@ import org.mockito.Mock;
import org.mockito.Spy;
import org.mockito.junit.MockitoJUnitRunner;
import java.net.UnknownHostException;
import java.nio.ByteBuffer;
import java.util.Set;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.when;
......@@ -61,17 +70,30 @@ public class AdminBrokerProcessorTest {
@Spy
private BrokerController
brokerController = new BrokerController(new BrokerConfig(), new NettyServerConfig(), new NettyClientConfig(),
new MessageStoreConfig());
brokerController = new BrokerController(new BrokerConfig(), new NettyServerConfig(), new NettyClientConfig(),
new MessageStoreConfig());
@Mock
private MessageStore messageStore;
private Set<String> systemTopicSet;
@Before
public void init() {
brokerController.setMessageStore(messageStore);
adminBrokerProcessor = new AdminBrokerProcessor(brokerController);
systemTopicSet = Sets.newHashSet(
TopicValidator.RMQ_SYS_SELF_TEST_TOPIC,
TopicValidator.RMQ_SYS_BENCHMARK_TOPIC,
TopicValidator.RMQ_SYS_SCHEDULE_TOPIC,
TopicValidator.RMQ_SYS_OFFSET_MOVED_EVENT,
TopicValidator.AUTO_CREATE_TOPIC_KEY_TOPIC,
this.brokerController.getBrokerConfig().getBrokerClusterName(),
this.brokerController.getBrokerConfig().getBrokerClusterName() + "_" + MixAll.REPLY_TOPIC_POSTFIX);
if (this.brokerController.getBrokerConfig().isTraceTopicEnable()) {
systemTopicSet.add(this.brokerController.getBrokerConfig().getMsgTraceTopicName());
}
}
@Test
......@@ -94,6 +116,67 @@ public class AdminBrokerProcessorTest {
assertThat(response.getCode()).isEqualTo(ResponseCode.SYSTEM_ERROR);
}
@Test
public void testUpdateAndCreateTopic() throws Exception {
//test system topic
for (String topic : systemTopicSet) {
RemotingCommand request = buildCreateTopicRequest(topic);
RemotingCommand response = adminBrokerProcessor.processRequest(handlerContext, request);
assertThat(response.getCode()).isEqualTo(ResponseCode.SYSTEM_ERROR);
assertThat(response.getRemark()).isEqualTo("The topic[" + topic + "] is conflict with system topic.");
}
//test validate error topic
String topic = "";
RemotingCommand request = buildCreateTopicRequest(topic);
RemotingCommand response = adminBrokerProcessor.processRequest(handlerContext, request);
assertThat(response.getCode()).isEqualTo(ResponseCode.SYSTEM_ERROR);
topic = "TEST_CREATE_TOPIC";
request = buildCreateTopicRequest(topic);
response = adminBrokerProcessor.processRequest(handlerContext, request);
assertThat(response.getCode()).isEqualTo(ResponseCode.SUCCESS);
}
@Test
public void testDeleteTopic() throws Exception {
//test system topic
for (String topic : systemTopicSet) {
RemotingCommand request = buildDeleteTopicRequest(topic);
RemotingCommand response = adminBrokerProcessor.processRequest(handlerContext, request);
assertThat(response.getCode()).isEqualTo(ResponseCode.SYSTEM_ERROR);
assertThat(response.getRemark()).isEqualTo("The topic[" + topic + "] is conflict with system topic.");
}
String topic = "TEST_DELETE_TOPIC";
RemotingCommand request = buildDeleteTopicRequest(topic);
RemotingCommand response = adminBrokerProcessor.processRequest(handlerContext, request);
assertThat(response.getCode()).isEqualTo(ResponseCode.SUCCESS);
}
private RemotingCommand buildCreateTopicRequest(String topic) {
CreateTopicRequestHeader requestHeader = new CreateTopicRequestHeader();
requestHeader.setTopic(topic);
requestHeader.setTopicFilterType(TopicFilterType.SINGLE_TAG.name());
requestHeader.setReadQueueNums(8);
requestHeader.setWriteQueueNums(8);
requestHeader.setPerm(PermName.PERM_READ | PermName.PERM_WRITE);
RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.UPDATE_AND_CREATE_TOPIC, requestHeader);
request.makeCustomHeaderToNet();
return request;
}
private RemotingCommand buildDeleteTopicRequest(String topic) {
DeleteTopicRequestHeader requestHeader = new DeleteTopicRequestHeader();
requestHeader.setTopic(topic);
RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.DELETE_TOPIC_IN_BROKER, requestHeader);
request.makeCustomHeaderToNet();
return request;
}
private MessageExt createDefaultMessageExt() {
MessageExt messageExt = new MessageExt();
messageExt.setMsgId("12345678");
......@@ -106,10 +189,11 @@ public class AdminBrokerProcessorTest {
return messageExt;
}
private SelectMappedBufferResult createSelectMappedBufferResult(){
SelectMappedBufferResult result = new SelectMappedBufferResult(0, ByteBuffer.allocate(1024) ,0, new MappedFile());
private SelectMappedBufferResult createSelectMappedBufferResult() {
SelectMappedBufferResult result = new SelectMappedBufferResult(0, ByteBuffer.allocate(1024), 0, new MappedFile());
return result;
}
private ResumeCheckHalfMessageRequestHeader createResumeCheckHalfMessageRequestHeader() {
ResumeCheckHalfMessageRequestHeader header = new ResumeCheckHalfMessageRequestHeader();
header.setMsgId("C0A803CA00002A9F0000000000031367");
......
broker/src/test/java/org/apache/rocketmq/broker/processor/ReplyMessageProcessorTest.java
浏览文件 @ 70295453
......@@ -25,15 +25,14 @@ import java.util.Map;
import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.broker.client.ClientChannelInfo;
import org.apache.rocketmq.broker.client.net.Broker2Client;
import org.apache.rocketmq.broker.transaction.TransactionalMessageService;
import org.apache.rocketmq.common.BrokerConfig;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageDecoder;
import org.apache.rocketmq.common.protocol.RequestCode;
import org.apache.rocketmq.common.protocol.ResponseCode;
import org.apache.rocketmq.common.protocol.header.SendMessageRequestHeader;
import org.apache.rocketmq.common.protocol.header.SendMessageResponseHeader;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
import org.apache.rocketmq.remoting.exception.RemotingSendRequestException;
import org.apache.rocketmq.remoting.exception.RemotingTimeoutException;
......@@ -115,7 +114,7 @@ public class ReplyMessageProcessorTest {
SendMessageRequestHeader requestHeader = new SendMessageRequestHeader();
requestHeader.setProducerGroup(group);
requestHeader.setTopic(topic);
requestHeader.setDefaultTopic(MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC);
requestHeader.setDefaultTopic(TopicValidator.AUTO_CREATE_TOPIC_KEY_TOPIC);
requestHeader.setDefaultTopicQueueNums(3);
requestHeader.setQueueId(1);
requestHeader.setSysFlag(0);
......
broker/src/test/java/org/apache/rocketmq/broker/processor/SendMessageProcessorTest.java
浏览文件 @ 70295453
......@@ -23,7 +23,6 @@ import org.apache.rocketmq.broker.mqtrace.SendMessageContext;
import org.apache.rocketmq.broker.mqtrace.SendMessageHook;
import org.apache.rocketmq.broker.transaction.TransactionalMessageService;
import org.apache.rocketmq.common.BrokerConfig;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageDecoder;
import org.apache.rocketmq.common.message.MessageExt;
......@@ -32,6 +31,7 @@ import org.apache.rocketmq.common.protocol.ResponseCode;
import org.apache.rocketmq.common.protocol.header.ConsumerSendMsgBackRequestHeader;
import org.apache.rocketmq.common.protocol.header.SendMessageRequestHeader;
import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.remoting.exception.RemotingCommandException;
import org.apache.rocketmq.remoting.netty.NettyClientConfig;
import org.apache.rocketmq.remoting.netty.NettyServerConfig;
......@@ -238,7 +238,7 @@ public class SendMessageProcessorTest {
SendMessageRequestHeader requestHeader = new SendMessageRequestHeader();
requestHeader.setProducerGroup(group);
requestHeader.setTopic(topic);
requestHeader.setDefaultTopic(MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC);
requestHeader.setDefaultTopic(TopicValidator.AUTO_CREATE_TOPIC_KEY_TOPIC);
requestHeader.setDefaultTopicQueueNums(3);
requestHeader.setQueueId(1);
requestHeader.setSysFlag(0);
......
broker/src/test/java/org/apache/rocketmq/broker/transaction/queue/DefaultTransactionalMessageCheckListenerTest.java
浏览文件 @ 70295453
......@@ -19,10 +19,10 @@ package org.apache.rocketmq.broker.transaction.queue;
import java.net.InetSocketAddress;
import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.common.BrokerConfig;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.message.MessageAccessor;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.remoting.netty.NettyClientConfig;
import org.apache.rocketmq.remoting.netty.NettyServerConfig;
import org.apache.rocketmq.store.MessageExtBrokerInner;
......@@ -92,7 +92,7 @@ public class DefaultTransactionalMessageCheckListenerTest {
@Test
public void testResolveDiscardMsg() {
MessageExt messageExt = new MessageExt();
messageExt.setTopic(MixAll.RMQ_SYS_TRANS_HALF_TOPIC);
messageExt.setTopic(TopicValidator.RMQ_SYS_TRANS_HALF_TOPIC);
messageExt.setQueueId(0);
messageExt.setBody("test resolve discard msg".getBytes());
messageExt.setStoreHost(new InetSocketAddress("127.0.0.1", 10911));
......
broker/src/test/java/org/apache/rocketmq/broker/transaction/queue/TransactionalMessageBridgeTest.java
浏览文件 @ 70295453
......@@ -20,11 +20,11 @@ import org.apache.rocketmq.broker.BrokerController;
import org.apache.rocketmq.client.consumer.PullResult;
import org.apache.rocketmq.client.consumer.PullStatus;
import org.apache.rocketmq.common.BrokerConfig;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.message.MessageAccessor;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.remoting.netty.NettyClientConfig;
import org.apache.rocketmq.remoting.netty.NettyServerConfig;
import org.apache.rocketmq.store.AppendMessageResult;
......@@ -98,7 +98,7 @@ public class TransactionalMessageBridgeTest {
@Test
public void testFetchMessageQueues() {
Set<MessageQueue> messageQueues = transactionBridge.fetchMessageQueues(MixAll.RMQ_SYS_TRANS_HALF_TOPIC);
Set<MessageQueue> messageQueues = transactionBridge.fetchMessageQueues(TopicValidator.RMQ_SYS_TRANS_HALF_TOPIC);
assertThat(messageQueues.size()).isEqualTo(1);
}
......
broker/src/test/java/org/apache/rocketmq/broker/transaction/queue/TransactionalMessageServiceImplTest.java
浏览文件 @ 70295453
......@@ -23,13 +23,13 @@ import org.apache.rocketmq.broker.transaction.TransactionalMessageService;
import org.apache.rocketmq.client.consumer.PullResult;
import org.apache.rocketmq.client.consumer.PullStatus;
import org.apache.rocketmq.common.BrokerConfig;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.protocol.ResponseCode;
import org.apache.rocketmq.common.protocol.header.EndTransactionRequestHeader;
import org.apache.rocketmq.common.sysflag.MessageSysFlag;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.remoting.netty.NettyClientConfig;
import org.apache.rocketmq.remoting.netty.NettyServerConfig;
import org.apache.rocketmq.store.AppendMessageResult;
......@@ -108,10 +108,10 @@ public class TransactionalMessageServiceImplTest {
@Test
public void testCheck_withDiscard() {
when(bridge.fetchMessageQueues(MixAll.RMQ_SYS_TRANS_HALF_TOPIC)).thenReturn(createMessageQueueSet(MixAll.RMQ_SYS_TRANS_HALF_TOPIC));
when(bridge.getHalfMessage(0, 0, 1)).thenReturn(createDiscardPullResult(MixAll.RMQ_SYS_TRANS_HALF_TOPIC, 5, "hellp", 1));
when(bridge.getHalfMessage(0, 1, 1)).thenReturn(createPullResult(MixAll.RMQ_SYS_TRANS_HALF_TOPIC, 6, "hellp", 0));
when(bridge.getOpMessage(anyInt(), anyLong(), anyInt())).thenReturn(createOpPulResult(MixAll.RMQ_SYS_TRANS_OP_HALF_TOPIC, 1, "10", 1));
when(bridge.fetchMessageQueues(TopicValidator.RMQ_SYS_TRANS_HALF_TOPIC)).thenReturn(createMessageQueueSet(TopicValidator.RMQ_SYS_TRANS_HALF_TOPIC));
when(bridge.getHalfMessage(0, 0, 1)).thenReturn(createDiscardPullResult(TopicValidator.RMQ_SYS_TRANS_HALF_TOPIC, 5, "hellp", 1));
when(bridge.getHalfMessage(0, 1, 1)).thenReturn(createPullResult(TopicValidator.RMQ_SYS_TRANS_HALF_TOPIC, 6, "hellp", 0));
when(bridge.getOpMessage(anyInt(), anyLong(), anyInt())).thenReturn(createOpPulResult(TopicValidator.RMQ_SYS_TRANS_OP_HALF_TOPIC, 1, "10", 1));
long timeOut = this.brokerController.getBrokerConfig().getTransactionTimeOut();
int checkMax = this.brokerController.getBrokerConfig().getTransactionCheckMax();
final AtomicInteger checkMessage = new AtomicInteger(0);
......@@ -128,10 +128,10 @@ public class TransactionalMessageServiceImplTest {
@Test
public void testCheck_withCheck() {
when(bridge.fetchMessageQueues(MixAll.RMQ_SYS_TRANS_HALF_TOPIC)).thenReturn(createMessageQueueSet(MixAll.RMQ_SYS_TRANS_HALF_TOPIC));
when(bridge.getHalfMessage(0, 0, 1)).thenReturn(createPullResult(MixAll.RMQ_SYS_TRANS_HALF_TOPIC, 5, "hello", 1));
when(bridge.getHalfMessage(0, 1, 1)).thenReturn(createPullResult(MixAll.RMQ_SYS_TRANS_HALF_TOPIC, 6, "hellp", 0));
when(bridge.getOpMessage(anyInt(), anyLong(), anyInt())).thenReturn(createPullResult(MixAll.RMQ_SYS_TRANS_OP_HALF_TOPIC, 1, "5", 0));
when(bridge.fetchMessageQueues(TopicValidator.RMQ_SYS_TRANS_HALF_TOPIC)).thenReturn(createMessageQueueSet(TopicValidator.RMQ_SYS_TRANS_HALF_TOPIC));
when(bridge.getHalfMessage(0, 0, 1)).thenReturn(createPullResult(TopicValidator.RMQ_SYS_TRANS_HALF_TOPIC, 5, "hello", 1));
when(bridge.getHalfMessage(0, 1, 1)).thenReturn(createPullResult(TopicValidator.RMQ_SYS_TRANS_HALF_TOPIC, 6, "hellp", 0));
when(bridge.getOpMessage(anyInt(), anyLong(), anyInt())).thenReturn(createPullResult(TopicValidator.RMQ_SYS_TRANS_OP_HALF_TOPIC, 1, "5", 0));
when(bridge.getBrokerController()).thenReturn(this.brokerController);
when(bridge.renewHalfMessageInner(any(MessageExtBrokerInner.class))).thenReturn(createMessageBrokerInner());
when(bridge.putMessageReturnResult(any(MessageExtBrokerInner.class))).thenReturn(new PutMessageResult
......
client/pom.xml
浏览文件 @ 70295453
......@@ -19,7 +19,7 @@
<parent>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-all</artifactId>
<version>4.7.1-SNAPSHOT</version>
<version>4.9.1-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
......@@ -27,11 +27,6 @@
<artifactId>rocketmq-client</artifactId>
<name>rocketmq-client ${project.version}</name>
<properties>
<maven.compiler.source>1.6</maven.compiler.source>
<maven.compiler.target>1.6</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
......@@ -52,6 +47,22 @@
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</dependency>
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
</dependency>
<dependency>
<groupId>io.opentracing</groupId>
<artifactId>opentracing-api</artifactId>
<version>0.33.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.opentracing</groupId>
<artifactId>opentracing-mock</artifactId>
<version>0.33.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
......
client/src/main/java/org/apache/rocketmq/client/ClientConfig.java
浏览文件 @ 70295453
......@@ -94,7 +94,7 @@ public class ClientConfig {
public void changeInstanceNameToPID() {
if (this.instanceName.equals("DEFAULT")) {
this.instanceName = String.valueOf(UtilAll.getPid());
this.instanceName = UtilAll.getPid() + "#" + System.nanoTime();
}
}
......@@ -178,8 +178,8 @@ public class ClientConfig {
}
public String getNamesrvAddr() {
if (StringUtils.isNotEmpty(namesrvAddr) && NameServerAddressUtils.NAMESRV_ENDPOINT_PATTERN.matcher(namesrvAddr.trim()).matches()) {
return namesrvAddr.substring(NameServerAddressUtils.ENDPOINT_PREFIX.length());
if (StringUtils.isNotEmpty(namesrvAddr) && NameServerAddressUtils.validateInstanceEndpoint(namesrvAddr.trim())) {
return NameServerAddressUtils.getNameSrvAddrFromNamesrvEndpoint(namesrvAddr);
}
return namesrvAddr;
}
......
client/src/main/java/org/apache/rocketmq/client/Validators.java
浏览文件 @ 70295453
......@@ -21,10 +21,10 @@ import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.client.producer.DefaultMQProducer;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.common.protocol.ResponseCode;
import org.apache.rocketmq.common.topic.TopicValidator;
/**
* Common Validator
......@@ -85,6 +85,7 @@ public class Validators {
}
// topic
Validators.checkTopic(msg.getTopic());
Validators.isNotAllowedSendTopic(msg.getTopic());
// body
if (null == msg.getBody()) {
......@@ -116,11 +117,19 @@ public class Validators {
throw new MQClientException(
String.format("The specified topic is longer than topic max length %d.", TOPIC_MAX_LENGTH), null);
}
}
public static void isSystemTopic(String topic) throws MQClientException {
if (TopicValidator.isSystemTopic(topic)) {
throw new MQClientException(
String.format("The topic[%s] is conflict with system topic.", topic), null);
}
}
//whether the same with system reserved keyword
if (topic.equals(MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC)) {
public static void isNotAllowedSendTopic(String topic) throws MQClientException {
if (TopicValidator.isNotAllowedSendTopic(topic)) {
throw new MQClientException(
String.format("The topic[%s] is conflict with AUTO_CREATE_TOPIC_KEY_TOPIC.", topic), null);
String.format("Sending message to topic[%s] is forbidden.", topic), null);
}
}
......
client/src/main/java/org/apache/rocketmq/client/common/ThreadLocalIndex.java
浏览文件 @ 70295453
......@@ -23,21 +23,15 @@ public class ThreadLocalIndex {
private final ThreadLocal<Integer> threadLocalIndex = new ThreadLocal<Integer>();
private final Random random = new Random();
public int getAndIncrement() {
public int incrementAndGet() {
Integer index = this.threadLocalIndex.get();
if (null == index) {
index = Math.abs(random.nextInt());
if (index < 0)
index = 0;
this.threadLocalIndex.set(index);
}
index = Math.abs(index + 1);
if (index < 0)
index = 0;
this.threadLocalIndex.set(index);
return index;
this.threadLocalIndex.set(++index);
return Math.abs(index);
}
@Override
......
client/src/main/java/org/apache/rocketmq/client/consumer/DefaultLitePullConsumer.java
浏览文件 @ 70295453
......@@ -23,6 +23,10 @@ import org.apache.rocketmq.client.consumer.rebalance.AllocateMessageQueueAverage
import org.apache.rocketmq.client.consumer.store.OffsetStore;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.client.impl.consumer.DefaultLitePullConsumerImpl;
import org.apache.rocketmq.client.log.ClientLogger;
import org.apache.rocketmq.client.trace.AsyncTraceDispatcher;
import org.apache.rocketmq.client.trace.TraceDispatcher;
import org.apache.rocketmq.client.trace.hook.ConsumeMessageTraceHookImpl;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
......@@ -30,10 +34,13 @@ import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.protocol.NamespaceUtil;
import org.apache.rocketmq.common.protocol.heartbeat.MessageModel;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.remoting.RPCHook;
public class DefaultLitePullConsumer extends ClientConfig implements LitePullConsumer {
private final InternalLogger log = ClientLogger.getLog();
private final DefaultLitePullConsumerImpl defaultLitePullConsumerImpl;
/**
......@@ -153,6 +160,21 @@ public class DefaultLitePullConsumer extends ClientConfig implements LitePullCon
*/
private String consumeTimestamp = UtilAll.timeMillisToHumanString3(System.currentTimeMillis() - (1000 * 60 * 30));
/**
* Interface of asynchronous transfer data
*/
private TraceDispatcher traceDispatcher = null;
/**
* The flag for message trace
*/
private boolean enableMsgTrace = false;
/**
* The name value of message trace topic.If you don't config,you can use the default trace topic name.
*/
private String customizedTraceTopic;
/**
* Default constructor.
*/
......@@ -202,13 +224,29 @@ public class DefaultLitePullConsumer extends ClientConfig implements LitePullCon
@Override
public void start() throws MQClientException {
setTraceDispatcher();
setConsumerGroup(NamespaceUtil.wrapNamespace(this.getNamespace(), this.consumerGroup));
this.defaultLitePullConsumerImpl.start();
if (null != traceDispatcher) {
try {
traceDispatcher.start(this.getNamesrvAddr(), this.getAccessChannel());
} catch (MQClientException e) {
log.warn("trace dispatcher start failed ", e);
}
}
}
@Override
public void shutdown() {
this.defaultLitePullConsumerImpl.shutdown();
if (null != traceDispatcher) {
traceDispatcher.shutdown();
}
}
@Override
public boolean isRunning() {
return this.defaultLitePullConsumerImpl.isRunning();
}
@Override
......@@ -307,6 +345,22 @@ public class DefaultLitePullConsumer extends ClientConfig implements LitePullCon
this.autoCommit = autoCommit;
}
public boolean isConnectBrokerByUser() {
return this.defaultLitePullConsumerImpl.getPullAPIWrapper().isConnectBrokerByUser();
}
public void setConnectBrokerByUser(boolean connectBrokerByUser) {
this.defaultLitePullConsumerImpl.getPullAPIWrapper().setConnectBrokerByUser(connectBrokerByUser);
}
public long getDefaultBrokerId() {
return this.defaultLitePullConsumerImpl.getPullAPIWrapper().getDefaultBrokerId();
}
public void setDefaultBrokerId(long defaultBrokerId) {
this.defaultLitePullConsumerImpl.getPullAPIWrapper().setDefaultBrokerId(defaultBrokerId);
}
public int getPullThreadNums() {
return pullThreadNums;
}
......@@ -469,4 +523,36 @@ public class DefaultLitePullConsumer extends ClientConfig implements LitePullCon
public void setConsumeTimestamp(String consumeTimestamp) {
this.consumeTimestamp = consumeTimestamp;
}
public TraceDispatcher getTraceDispatcher() {
return traceDispatcher;
}
public void setCustomizedTraceTopic(String customizedTraceTopic) {
this.customizedTraceTopic = customizedTraceTopic;
}
private void setTraceDispatcher() {
if (isEnableMsgTrace()) {
try {
this.traceDispatcher = new AsyncTraceDispatcher(consumerGroup, TraceDispatcher.Type.CONSUME, customizedTraceTopic, null);
this.defaultLitePullConsumerImpl.registerConsumeMessageHook(
new ConsumeMessageTraceHookImpl(traceDispatcher));
} catch (Throwable e) {
log.error("system mqtrace hook init failed ,maybe can't send msg trace data");
}
}
}
public String getCustomizedTraceTopic() {
return customizedTraceTopic;
}
public boolean isEnableMsgTrace() {
return enableMsgTrace;
}
public void setEnableMsgTrace(boolean enableMsgTrace) {
this.enableMsgTrace = enableMsgTrace;
}
}
client/src/main/java/org/apache/rocketmq/client/consumer/DefaultMQPushConsumer.java
浏览文件 @ 70295453
......@@ -254,6 +254,11 @@ public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsume
*/
private long consumeTimeout = 15;
/**
* Maximum time to await message consuming when shutdown consumer, 0 indicates no await.
*/
private long awaitTerminationMillisWhenShutdown = 0;
/**
* Interface of asynchronous transfer data
*/
......@@ -705,7 +710,7 @@ public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsume
*/
@Override
public void shutdown() {
this.defaultMQPushConsumerImpl.shutdown();
this.defaultMQPushConsumerImpl.shutdown(awaitTerminationMillisWhenShutdown);
if (null != traceDispatcher) {
traceDispatcher.shutdown();
}
......@@ -886,6 +891,14 @@ public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsume
this.consumeTimeout = consumeTimeout;
}
public long getAwaitTerminationMillisWhenShutdown() {
return awaitTerminationMillisWhenShutdown;
}
public void setAwaitTerminationMillisWhenShutdown(long awaitTerminationMillisWhenShutdown) {
this.awaitTerminationMillisWhenShutdown = awaitTerminationMillisWhenShutdown;
}
public TraceDispatcher getTraceDispatcher() {
return traceDispatcher;
}
......
client/src/main/java/org/apache/rocketmq/client/consumer/LitePullConsumer.java
浏览文件 @ 70295453
......@@ -35,6 +35,13 @@ public interface LitePullConsumer {
*/
void shutdown();
/**
* This consumer is still running
*
* @return true if consumer is still running
*/
boolean isRunning();
/**
* Subscribe some topic with subExpression
*
......
client/src/main/java/org/apache/rocketmq/client/consumer/rebalance/AllocateMessageQueueByMachineRoom.java
浏览文件 @ 70295453
......@@ -49,7 +49,7 @@ public class AllocateMessageQueueByMachineRoom implements AllocateMessageQueueSt
int startIndex = mod * currentIndex;
int endIndex = startIndex + mod;
for (int i = startIndex; i < endIndex; i++) {
result.add(mqAll.get(i));
result.add(premqAll.get(i));
}
if (rem > currentIndex) {
result.add(premqAll.get(currentIndex + mod * cidAll.size()));
......
client/src/main/java/org/apache/rocketmq/client/exception/MQBrokerException.java
浏览文件 @ 70295453
......@@ -23,12 +23,22 @@ public class MQBrokerException extends Exception {
private static final long serialVersionUID = 5975020272601250368L;
private final int responseCode;
private final String errorMessage;
private final String brokerAddr;
public MQBrokerException(int responseCode, String errorMessage) {
super(FAQUrl.attachDefaultURL("CODE: " + UtilAll.responseCode2String(responseCode) + " DESC: "
+ errorMessage));
+ errorMessage));
this.responseCode = responseCode;
this.errorMessage = errorMessage;
this.brokerAddr = null;
}
public MQBrokerException(int responseCode, String errorMessage, String brokerAddr) {
super(FAQUrl.attachDefaultURL("CODE: " + UtilAll.responseCode2String(responseCode) + " DESC: "
+ errorMessage + (brokerAddr != null ? " BROKER: " + brokerAddr : "")));
this.responseCode = responseCode;
this.errorMessage = errorMessage;
this.brokerAddr = brokerAddr;
}
public int getResponseCode() {
......@@ -38,4 +48,8 @@ public class MQBrokerException extends Exception {
public String getErrorMessage() {
return errorMessage;
}
public String getBrokerAddr() {
return brokerAddr;
}
}
client/src/main/java/org/apache/rocketmq/client/hook/EndTransactionContext.java 0 → 100644
浏览文件 @ 70295453
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.rocketmq.client.hook;
import org.apache.rocketmq.client.producer.LocalTransactionState;
import org.apache.rocketmq.common.message.Message;
public class EndTransactionContext {
private String producerGroup;
private Message message;
private String brokerAddr;
private String msgId;
private String transactionId;
private LocalTransactionState transactionState;
private boolean fromTransactionCheck;
public String getProducerGroup() {
return producerGroup;
}
public void setProducerGroup(String producerGroup) {
this.producerGroup = producerGroup;
}
public Message getMessage() {
return message;
}
public void setMessage(Message message) {
this.message = message;
}
public String getBrokerAddr() {
return brokerAddr;
}
public void setBrokerAddr(String brokerAddr) {
this.brokerAddr = brokerAddr;
}
public String getMsgId() {
return msgId;
}
public void setMsgId(String msgId) {
this.msgId = msgId;
}
public String getTransactionId() {
return transactionId;
}
public void setTransactionId(String transactionId) {
this.transactionId = transactionId;
}
public LocalTransactionState getTransactionState() {
return transactionState;
}
public void setTransactionState(LocalTransactionState transactionState) {
this.transactionState = transactionState;
}
public boolean isFromTransactionCheck() {
return fromTransactionCheck;
}
public void setFromTransactionCheck(boolean fromTransactionCheck) {
this.fromTransactionCheck = fromTransactionCheck;
}
}
client/src/main/java/org/apache/rocketmq/client/hook/EndTransactionHook.java 0 → 100644
浏览文件 @ 70295453
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.rocketmq.client.hook;
public interface EndTransactionHook {
String hookName();
void endTransaction(final EndTransactionContext context);
}
client/src/main/java/org/apache/rocketmq/client/impl/ClientRemotingProcessor.java
浏览文件 @ 70295453
......@@ -282,8 +282,6 @@ public class ClientRemotingProcessor extends AsyncNettyRequestProcessor implemen
if (requestResponseFuture.getRequestCallback() != null) {
requestResponseFuture.getRequestCallback().onSuccess(replyMsg);
} else {
requestResponseFuture.putResponseMessage(replyMsg);
}
} else {
String bornHost = replyMsg.getBornHostString();
......
client/src/main/java/org/apache/rocketmq/client/impl/MQAdminImpl.java
浏览文件 @ 70295453
......@@ -82,6 +82,7 @@ public class MQAdminImpl {
public void createTopic(String key, String newTopic, int queueNum, int topicSysFlag) throws MQClientException {
try {
Validators.checkTopic(newTopic);
Validators.isSystemTopic(newTopic);
TopicRouteData topicRouteData = this.mQClientFactory.getMQClientAPIImpl().getTopicRouteInfoFromNameServer(key, timeoutMillis);
List<BrokerData> brokerDataList = topicRouteData.getBrokerDatas();
if (brokerDataList != null && !brokerDataList.isEmpty()) {
......@@ -267,17 +268,23 @@ public class MQAdminImpl {
messageId.getOffset(), timeoutMillis);
}
public QueryResult queryMessage(String topic, String key, int maxNum, long begin,
long end) throws MQClientException,
InterruptedException {
public QueryResult queryMessage(String topic, String key, int maxNum, long begin, long end)
throws MQClientException, InterruptedException {
return queryMessage(topic, key, maxNum, begin, end, false);
}
public QueryResult queryMessageByUniqKey(String topic, String uniqKey, int maxNum, long begin, long end)
throws MQClientException, InterruptedException {
return queryMessage(topic, uniqKey, maxNum, begin, end, true);
}
public MessageExt queryMessageByUniqKey(String topic,
String uniqKey) throws InterruptedException, MQClientException {
QueryResult qr = this.queryMessage(topic, uniqKey, 32,
MessageClientIDSetter.getNearlyTimeFromID(uniqKey).getTime() - 1000, Long.MAX_VALUE, true);
QueryResult qr = queryMessageByUniqKey(topic, uniqKey, 32,
MessageClientIDSetter.getNearlyTimeFromID(uniqKey).getTime() - 1000, Long.MAX_VALUE);
if (qr != null && qr.getMessageList() != null && qr.getMessageList().size() > 0) {
return qr.getMessageList().get(0);
} else {
......
client/src/main/java/org/apache/rocketmq/client/impl/MQClientAPIImpl.java
浏览文件 @ 70295453
......@@ -390,7 +390,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
......@@ -414,7 +414,7 @@ public class MQClientAPIImpl {
default:
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
......@@ -502,7 +502,7 @@ public class MQClientAPIImpl {
) throws RemotingException, MQBrokerException, InterruptedException {
RemotingCommand response = this.remotingClient.invokeSync(addr, request, timeoutMillis);
assert response != null;
return this.processSendResponse(brokerName, msg, response);
return this.processSendResponse(brokerName, msg, response,addr);
}
private void sendMessageAsync(
......@@ -528,7 +528,7 @@ public class MQClientAPIImpl {
if (null == sendCallback && response != null) {
try {
SendResult sendResult = MQClientAPIImpl.this.processSendResponse(brokerName, msg, response);
SendResult sendResult = MQClientAPIImpl.this.processSendResponse(brokerName, msg, response, addr);
if (context != null && sendResult != null) {
context.setSendResult(sendResult);
context.getProducer().executeSendMessageHookAfter(context);
......@@ -542,7 +542,7 @@ public class MQClientAPIImpl {
if (response != null) {
try {
SendResult sendResult = MQClientAPIImpl.this.processSendResponse(brokerName, msg, response);
SendResult sendResult = MQClientAPIImpl.this.processSendResponse(brokerName, msg, response, addr);
assert sendResult != null;
if (context != null) {
context.setSendResult(sendResult);
......@@ -641,7 +641,8 @@ public class MQClientAPIImpl {
private SendResult processSendResponse(
final String brokerName,
final Message msg,
final RemotingCommand response
final RemotingCommand response,
final String addr
) throws MQBrokerException, RemotingCommandException {
SendStatus sendStatus;
switch (response.getCode()) {
......@@ -662,7 +663,7 @@ public class MQClientAPIImpl {
break;
}
default: {
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
}
......@@ -741,7 +742,7 @@ public class MQClientAPIImpl {
RemotingCommand response = responseFuture.getResponseCommand();
if (response != null) {
try {
PullResult pullResult = MQClientAPIImpl.this.processPullResponse(response);
PullResult pullResult = MQClientAPIImpl.this.processPullResponse(response, addr);
assert pullResult != null;
pullCallback.onSuccess(pullResult);
} catch (Exception e) {
......@@ -768,11 +769,12 @@ public class MQClientAPIImpl {
) throws RemotingException, InterruptedException, MQBrokerException {
RemotingCommand response = this.remotingClient.invokeSync(addr, request, timeoutMillis);
assert response != null;
return this.processPullResponse(response);
return this.processPullResponse(response, addr);
}
private PullResult processPullResponse(
final RemotingCommand response) throws MQBrokerException, RemotingCommandException {
final RemotingCommand response,
final String addr) throws MQBrokerException, RemotingCommandException {
PullStatus pullStatus = PullStatus.NO_NEW_MSG;
switch (response.getCode()) {
case ResponseCode.SUCCESS:
......@@ -789,7 +791,7 @@ public class MQClientAPIImpl {
break;
default:
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
PullMessageResponseHeader responseHeader =
......@@ -822,7 +824,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public long searchOffset(final String addr, final String topic, final int queueId, final long timestamp,
......@@ -847,7 +849,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public long getMaxOffset(final String addr, final String topic, final int queueId, final long timeoutMillis)
......@@ -871,7 +873,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public List<String> getConsumerIdListByGroup(
......@@ -898,7 +900,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public long getMinOffset(final String addr, final String topic, final int queueId, final long timeoutMillis)
......@@ -922,7 +924,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public long getEarliestMsgStoretime(final String addr, final String topic, final int queueId,
......@@ -947,7 +949,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public long queryConsumerOffset(
......@@ -971,7 +973,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public void updateConsumerOffset(
......@@ -992,7 +994,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public void updateConsumerOffsetOneway(
......@@ -1024,7 +1026,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public void unregisterClient(
......@@ -1050,7 +1052,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public void endTransactionOneway(
......@@ -1116,7 +1118,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public Set<MessageQueue> lockBatchMQ(
......@@ -1138,7 +1140,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public void unlockBatchMQ(
......@@ -1164,7 +1166,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
}
......@@ -1187,7 +1189,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public ConsumeStats getConsumeStats(final String addr, final String consumerGroup, final long timeoutMillis)
......@@ -1217,7 +1219,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public ProducerConnection getProducerConnectionList(final String addr, final String producerGroup,
......@@ -1239,7 +1241,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public ConsumerConnection getConsumerConnectionList(final String addr, final String consumerGroup,
......@@ -1261,7 +1263,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public KVTable getBrokerRuntimeInfo(final String addr, final long timeoutMillis) throws RemotingConnectException,
......@@ -1279,7 +1281,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public void updateBrokerConfig(final String addr, final Properties properties, final long timeoutMillis)
......@@ -1301,7 +1303,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
}
......@@ -1320,7 +1322,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public ClusterInfo getBrokerClusterInfo(
......@@ -1364,7 +1366,7 @@ public class MQClientAPIImpl {
assert response != null;
switch (response.getCode()) {
case ResponseCode.TOPIC_NOT_EXIST: {
if (allowTopicNotExist && !topic.equals(MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC)) {
if (allowTopicNotExist) {
log.warn("get Topic [{}] RouteInfoFromNameServer is not exist value", topic);
}
......@@ -1465,10 +1467,11 @@ public class MQClientAPIImpl {
throw new MQClientException(response.getCode(), response.getRemark());
}
public void deleteSubscriptionGroup(final String addr, final String groupName, final long timeoutMillis)
public void deleteSubscriptionGroup(final String addr, final String groupName, final boolean removeOffset, final long timeoutMillis)
throws RemotingException, MQBrokerException, InterruptedException, MQClientException {
DeleteSubscriptionGroupRequestHeader requestHeader = new DeleteSubscriptionGroupRequestHeader();
requestHeader.setGroupName(groupName);
requestHeader.setRemoveOffset(removeOffset);
RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.DELETE_SUBSCRIPTIONGROUP, requestHeader);
RemotingCommand response = this.remotingClient.invokeSync(MixAll.brokerVIPChannel(this.clientConfig.isVipChannelEnabled(), addr),
......@@ -1670,7 +1673,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public List<QueueTimeSpan> queryConsumeTimeSpan(final String addr, final String topic, final String group,
......@@ -1694,7 +1697,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public TopicList getTopicsByCluster(final String cluster, final long timeoutMillis)
......@@ -1745,7 +1748,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public TopicList getSystemTopicList(
......@@ -2108,7 +2111,7 @@ public class MQClientAPIImpl {
default:
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), brokerAddr);
}
public TopicConfigSerializeWrapper getAllTopicConfig(final String addr,
......@@ -2127,7 +2130,7 @@ public class MQClientAPIImpl {
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark());
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
public void updateNameServerConfig(final Properties properties, final List<String> nameServers, long timeoutMillis)
......
client/src/main/java/org/apache/rocketmq/client/impl/consumer/AssignedMessageQueue.java
浏览文件 @ 70295453
......@@ -83,9 +83,12 @@ public class AssignedMessageQueue {
return -1;
}
public void updatePullOffset(MessageQueue messageQueue, long offset) {
public void updatePullOffset(MessageQueue messageQueue, long offset, ProcessQueue processQueue) {
MessageQueueState messageQueueState = assignedMessageQueueState.get(messageQueue);
if (messageQueueState != null) {
if (messageQueueState.getProcessQueue() != processQueue) {
return;
}
messageQueueState.setPullOffset(offset);
}
}
......
client/src/main/java/org/apache/rocketmq/client/impl/consumer/ConsumeMessageConcurrentlyService.java
浏览文件 @ 70295453
......@@ -45,6 +45,7 @@ import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.protocol.body.CMResult;
import org.apache.rocketmq.common.protocol.body.ConsumeMessageDirectlyResult;
import org.apache.rocketmq.common.utils.ThreadUtils;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.remoting.common.RemotingHelper;
......@@ -92,9 +93,9 @@ public class ConsumeMessageConcurrentlyService implements ConsumeMessageService
}, this.defaultMQPushConsumer.getConsumeTimeout(), this.defaultMQPushConsumer.getConsumeTimeout(), TimeUnit.MINUTES);
}
public void shutdown() {
public void shutdown(long awaitTerminateMillis) {
this.scheduledExecutorService.shutdown();
this.consumeExecutor.shutdown();
ThreadUtils.shutdownGracefully(this.consumeExecutor, awaitTerminateMillis, TimeUnit.MILLISECONDS);
this.cleanExpireMsgExecutors.shutdown();
}
......
client/src/main/java/org/apache/rocketmq/client/impl/consumer/ConsumeMessageOrderlyService.java
浏览文件 @ 70295453
......@@ -40,6 +40,7 @@ import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.ThreadFactoryImpl;
import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.protocol.NamespaceUtil;
import org.apache.rocketmq.common.utils.ThreadUtils;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.common.message.MessageAccessor;
......@@ -96,10 +97,10 @@ public class ConsumeMessageOrderlyService implements ConsumeMessageService {
}
}
public void shutdown() {
public void shutdown(long awaitTerminateMillis) {
this.stopped = true;
this.scheduledExecutorService.shutdown();
this.consumeExecutor.shutdown();
ThreadUtils.shutdownGracefully(this.consumeExecutor, awaitTerminateMillis, TimeUnit.MILLISECONDS);
if (MessageModel.CLUSTERING.equals(this.defaultMQPushConsumerImpl.messageModel())) {
this.unlockAllMQ();
}
......@@ -279,7 +280,7 @@ public class ConsumeMessageOrderlyService implements ConsumeMessageService {
case SUSPEND_CURRENT_QUEUE_A_MOMENT:
this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), msgs.size());
if (checkReconsumeTimes(msgs)) {
consumeRequest.getProcessQueue().makeMessageToCosumeAgain(msgs);
consumeRequest.getProcessQueue().makeMessageToConsumeAgain(msgs);
this.submitConsumeRequestLater(
consumeRequest.getProcessQueue(),
consumeRequest.getMessageQueue(),
......@@ -311,7 +312,7 @@ public class ConsumeMessageOrderlyService implements ConsumeMessageService {
case SUSPEND_CURRENT_QUEUE_A_MOMENT:
this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), msgs.size());
if (checkReconsumeTimes(msgs)) {
consumeRequest.getProcessQueue().makeMessageToCosumeAgain(msgs);
consumeRequest.getProcessQueue().makeMessageToConsumeAgain(msgs);
this.submitConsumeRequestLater(
consumeRequest.getProcessQueue(),
consumeRequest.getMessageQueue(),
......@@ -477,7 +478,7 @@ public class ConsumeMessageOrderlyService implements ConsumeMessageService {
ConsumeReturnType returnType = ConsumeReturnType.SUCCESS;
boolean hasException = false;
try {
this.processQueue.getLockConsume().lock();
this.processQueue.getConsumeLock().lock();
if (this.processQueue.isDropped()) {
log.warn("consumeMessage, the message queue not be able to consume, because it's dropped. {}",
this.messageQueue);
......@@ -493,7 +494,7 @@ public class ConsumeMessageOrderlyService implements ConsumeMessageService {
messageQueue);
hasException = true;
} finally {
this.processQueue.getLockConsume().unlock();
this.processQueue.getConsumeLock().unlock();
}
if (null == status
......
client/src/main/java/org/apache/rocketmq/client/impl/consumer/ConsumeMessageService.java
浏览文件 @ 70295453
......@@ -24,7 +24,7 @@ import org.apache.rocketmq.common.protocol.body.ConsumeMessageDirectlyResult;
public interface ConsumeMessageService {
void start();
void shutdown();
void shutdown(long awaitTerminateMillis);
void updateCorePoolSize(int corePoolSize);
......
client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultLitePullConsumerImpl.java
浏览文件 @ 70295453
......@@ -41,12 +41,15 @@ import org.apache.rocketmq.client.consumer.MessageQueueListener;
import org.apache.rocketmq.client.consumer.MessageSelector;
import org.apache.rocketmq.client.consumer.PullResult;
import org.apache.rocketmq.client.consumer.TopicMessageQueueChangeListener;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import org.apache.rocketmq.client.consumer.store.LocalFileOffsetStore;
import org.apache.rocketmq.client.consumer.store.OffsetStore;
import org.apache.rocketmq.client.consumer.store.ReadOffsetType;
import org.apache.rocketmq.client.consumer.store.RemoteBrokerOffsetStore;
import org.apache.rocketmq.client.exception.MQBrokerException;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.client.hook.ConsumeMessageContext;
import org.apache.rocketmq.client.hook.ConsumeMessageHook;
import org.apache.rocketmq.client.hook.FilterMessageHook;
import org.apache.rocketmq.client.impl.CommunicationMode;
import org.apache.rocketmq.client.impl.MQClientManager;
......@@ -115,6 +118,8 @@ public class DefaultLitePullConsumerImpl implements MQConsumerInner {
*/
private static final long PULL_TIME_DELAY_MILLS_WHEN_PAUSE = 1000;
private static final long PULL_TIME_DELAY_MILLS_ON_EXCEPTION = 3 * 1000;
private DefaultLitePullConsumer defaultLitePullConsumer;
private final ConcurrentMap<MessageQueue, PullTaskImpl> taskTable =
......@@ -142,6 +147,8 @@ public class DefaultLitePullConsumerImpl implements MQConsumerInner {
private final MessageQueueLock messageQueueLock = new MessageQueueLock();
private final ArrayList<ConsumeMessageHook> consumeMessageHookList = new ArrayList<>();
public DefaultLitePullConsumerImpl(final DefaultLitePullConsumer defaultLitePullConsumer, final RPCHook rpcHook) {
this.defaultLitePullConsumer = defaultLitePullConsumer;
this.rpcHook = rpcHook;
......@@ -158,6 +165,35 @@ public class DefaultLitePullConsumerImpl implements MQConsumerInner {
this.pullTimeDelayMillsWhenException = defaultLitePullConsumer.getPullTimeDelayMillsWhenException();
}
public void registerConsumeMessageHook(final ConsumeMessageHook hook) {
this.consumeMessageHookList.add(hook);
log.info("register consumeMessageHook Hook, {}", hook.hookName());
}
public void executeHookBefore(final ConsumeMessageContext context) {
if (!this.consumeMessageHookList.isEmpty()) {
for (ConsumeMessageHook hook : this.consumeMessageHookList) {
try {
hook.consumeMessageBefore(context);
} catch (Throwable e) {
log.error("consumeMessageHook {} executeHookBefore exception", hook.hookName(), e);
}
}
}
}
public void executeHookAfter(final ConsumeMessageContext context) {
if (!this.consumeMessageHookList.isEmpty()) {
for (ConsumeMessageHook hook : this.consumeMessageHookList) {
try {
hook.consumeMessageAfter(context);
} catch (Throwable e) {
log.error("consumeMessageHook {} executeHookAfter exception", hook.hookName(), e);
}
}
}
}
private void checkServiceState() {
if (this.serviceState != ServiceState.RUNNING)
throw new IllegalStateException(NOT_RUNNING_EXCEPTION_MESSAGE);
......@@ -229,6 +265,10 @@ public class DefaultLitePullConsumerImpl implements MQConsumerInner {
}
}
public synchronized boolean isRunning() {
return this.serviceState == ServiceState.RUNNING;
}
public synchronized void start() throws MQClientException {
switch (this.serviceState) {
case CREATE_JUST:
......@@ -428,8 +468,7 @@ public class DefaultLitePullConsumerImpl implements MQConsumerInner {
throw new IllegalArgumentException("Topic can not be null or empty.");
}
setSubscriptionType(SubscriptionType.SUBSCRIBE);
SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(defaultLitePullConsumer.getConsumerGroup(),
topic, subExpression);
SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(topic, subExpression);
this.rebalanceImpl.getSubscriptionInner().put(topic, subscriptionData);
this.defaultLitePullConsumer.setMessageQueueListener(new MessageQueueListenerImpl());
assignedMessageQueue.setRebalanceImpl(this.rebalanceImpl);
......@@ -609,9 +648,9 @@ public class DefaultLitePullConsumerImpl implements MQConsumerInner {
}
}
private void updatePullOffset(MessageQueue messageQueue, long nextPullOffset) {
private void updatePullOffset(MessageQueue messageQueue, long nextPullOffset, ProcessQueue processQueue) {
if (assignedMessageQueue.getSeekOffset(messageQueue) == -1) {
assignedMessageQueue.updatePullOffset(messageQueue, nextPullOffset);
assignedMessageQueue.updatePullOffset(messageQueue, nextPullOffset, processQueue);
}
}
......@@ -623,9 +662,9 @@ public class DefaultLitePullConsumerImpl implements MQConsumerInner {
}
}
private long fetchConsumeOffset(MessageQueue messageQueue) {
private long fetchConsumeOffset(MessageQueue messageQueue) throws MQClientException {
checkServiceState();
long offset = this.rebalanceImpl.computePullFromWhere(messageQueue);
long offset = this.rebalanceImpl.computePullFromWhereWithException(messageQueue);
return offset;
}
......@@ -649,7 +688,7 @@ public class DefaultLitePullConsumerImpl implements MQConsumerInner {
}
}
private long nextPullOffset(MessageQueue messageQueue) {
private long nextPullOffset(MessageQueue messageQueue) throws MQClientException {
long offset = -1;
long seekOffset = assignedMessageQueue.getSeekOffset(messageQueue);
if (seekOffset != -1) {
......@@ -691,7 +730,7 @@ public class DefaultLitePullConsumerImpl implements MQConsumerInner {
ProcessQueue processQueue = assignedMessageQueue.getProcessQueue(messageQueue);
if (processQueue == null && processQueue.isDropped()) {
if (null == processQueue || processQueue.isDropped()) {
log.info("The message queue not be able to poll, because it's dropped. group={}, messageQueue={}", defaultLitePullConsumer.getConsumerGroup(), this.messageQueue);
return;
}
......@@ -736,7 +775,18 @@ public class DefaultLitePullConsumerImpl implements MQConsumerInner {
return;
}
long offset = nextPullOffset(messageQueue);
long offset = 0L;
try {
offset = nextPullOffset(messageQueue);
} catch (MQClientException e) {
log.error("Failed to get next pull offset", e);
scheduledThreadPoolExecutor.schedule(this, PULL_TIME_DELAY_MILLS_ON_EXCEPTION, TimeUnit.MILLISECONDS);
return;
}
if (this.isCancelled() || processQueue.isDropped()) {
return;
}
long pullDelayTimeMills = 0;
try {
SubscriptionData subscriptionData;
......@@ -745,12 +795,13 @@ public class DefaultLitePullConsumerImpl implements MQConsumerInner {
subscriptionData = rebalanceImpl.getSubscriptionInner().get(topic);
} else {
String topic = this.messageQueue.getTopic();
subscriptionData = FilterAPI.buildSubscriptionData(defaultLitePullConsumer.getConsumerGroup(),
topic, SubscriptionData.SUB_ALL);
subscriptionData = FilterAPI.buildSubscriptionData(topic, SubscriptionData.SUB_ALL);
}
PullResult pullResult = pull(messageQueue, subscriptionData, offset, defaultLitePullConsumer.getPullBatchSize());
if (this.isCancelled() || processQueue.isDropped()) {
return;
}
switch (pullResult.getPullStatus()) {
case FOUND:
final Object objLock = messageQueueLock.fetchLockObject(messageQueue);
......@@ -767,7 +818,7 @@ public class DefaultLitePullConsumerImpl implements MQConsumerInner {
default:
break;
}
updatePullOffset(messageQueue, pullResult.getNextBeginOffset());
updatePullOffset(messageQueue, pullResult.getNextBeginOffset(), processQueue);
} catch (Throwable e) {
pullDelayTimeMills = pullTimeDelayMillsWhenException;
log.error("An error occurred in pull message process.", e);
......@@ -841,6 +892,18 @@ public class DefaultLitePullConsumerImpl implements MQConsumerInner {
null
);
this.pullAPIWrapper.processPullResult(mq, pullResult, subscriptionData);
if (!this.consumeMessageHookList.isEmpty()) {
ConsumeMessageContext consumeMessageContext = new ConsumeMessageContext();
consumeMessageContext.setNamespace(defaultLitePullConsumer.getNamespace());
consumeMessageContext.setConsumerGroup(this.groupName());
consumeMessageContext.setMq(mq);
consumeMessageContext.setMsgList(pullResult.getMsgFoundList());
consumeMessageContext.setSuccess(false);
this.executeHookBefore(consumeMessageContext);
consumeMessageContext.setStatus(ConsumeConcurrentlyStatus.CONSUME_SUCCESS.toString());
consumeMessageContext.setSuccess(true);
this.executeHookAfter(consumeMessageContext);
}
return pullResult;
}
......
client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPullConsumerImpl.java
浏览文件 @ 70295453
......@@ -205,8 +205,7 @@ public class DefaultMQPullConsumerImpl implements MQConsumerInner {
}
try {
return FilterAPI.buildSubscriptionData(this.defaultMQPullConsumer.getConsumerGroup(),
mq.getTopic(), subExpression);
return FilterAPI.buildSubscriptionData(mq.getTopic(), subExpression);
} catch (Exception e) {
throw new MQClientException("parse subscription error", e);
}
......@@ -301,8 +300,7 @@ public class DefaultMQPullConsumerImpl implements MQConsumerInner {
public void subscriptionAutomatically(final String topic) {
if (!this.rebalanceImpl.getSubscriptionInner().containsKey(topic)) {
try {
SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(this.defaultMQPullConsumer.getConsumerGroup(),
topic, SubscriptionData.SUB_ALL);
SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(topic, SubscriptionData.SUB_ALL);
this.rebalanceImpl.subscriptionInner.putIfAbsent(topic, subscriptionData);
} catch (Exception ignore) {
}
......@@ -365,7 +363,7 @@ public class DefaultMQPullConsumerImpl implements MQConsumerInner {
for (String t : topics) {
SubscriptionData ms = null;
try {
ms = FilterAPI.buildSubscriptionData(this.groupName(), t, SubscriptionData.SUB_ALL);
ms = FilterAPI.buildSubscriptionData(t, SubscriptionData.SUB_ALL);
} catch (Exception e) {
log.error("parse subscription error", e);
}
......@@ -742,8 +740,7 @@ public class DefaultMQPullConsumerImpl implements MQConsumerInner {
Set<String> registerTopics = this.defaultMQPullConsumer.getRegisterTopics();
if (registerTopics != null) {
for (final String topic : registerTopics) {
SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(this.defaultMQPullConsumer.getConsumerGroup(),
topic, SubscriptionData.SUB_ALL);
SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(topic, SubscriptionData.SUB_ALL);
this.rebalanceImpl.getSubscriptionInner().put(topic, subscriptionData);
}
}
......
client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java
浏览文件 @ 70295453
......@@ -269,8 +269,15 @@ public class DefaultMQPushConsumerImpl implements MQConsumerInner {
}
} else {
if (processQueue.isLocked()) {
if (!pullRequest.isLockedFirst()) {
final long offset = this.rebalanceImpl.computePullFromWhere(pullRequest.getMessageQueue());
if (!pullRequest.isPreviouslyLocked()) {
long offset = -1L;
try {
offset = this.rebalanceImpl.computePullFromWhereWithException(pullRequest.getMessageQueue());
} catch (MQClientException e) {
this.executePullRequestLater(pullRequest, pullTimeDelayMillsWhenException);
log.error("Failed to compute pull offset, pullResult: {}", pullRequest, e);
return;
}
boolean brokerBusy = offset < pullRequest.getNextOffset();
log.info("the first time to pull message, so fix offset from broker. pullRequest: {} NewOffset: {} brokerBusy: {}",
pullRequest, offset, brokerBusy);
......@@ -279,7 +286,7 @@ public class DefaultMQPushConsumerImpl implements MQConsumerInner {
pullRequest, offset);
}
pullRequest.setLockedFirst(true);
pullRequest.setPreviouslyLocked(true);
pullRequest.setNextOffset(offset);
}
} else {
......@@ -348,12 +355,6 @@ public class DefaultMQPushConsumerImpl implements MQConsumerInner {
break;
case NO_NEW_MSG:
pullRequest.setNextOffset(pullResult.getNextBeginOffset());
DefaultMQPushConsumerImpl.this.correctTagsOffset(pullRequest);
DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
break;
case NO_MATCHED_MSG:
pullRequest.setNextOffset(pullResult.getNextBeginOffset());
......@@ -546,12 +547,16 @@ public class DefaultMQPushConsumerImpl implements MQConsumerInner {
}
}
public synchronized void shutdown() {
public void shutdown() {
shutdown(0);
}
public synchronized void shutdown(long awaitTerminateMillis) {
switch (this.serviceState) {
case CREATE_JUST:
break;
case RUNNING:
this.consumeMessageService.shutdown();
this.consumeMessageService.shutdown(awaitTerminateMillis);
this.persistConsumerOffset();
this.mQClientFactory.unregisterConsumer(this.defaultMQPushConsumer.getConsumerGroup());
this.mQClientFactory.shutdown();
......@@ -625,7 +630,7 @@ public class DefaultMQPushConsumerImpl implements MQConsumerInner {
boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPushConsumer.getConsumerGroup(), this);
if (!registerOK) {
this.serviceState = ServiceState.CREATE_JUST;
this.consumeMessageService.shutdown();
this.consumeMessageService.shutdown(defaultMQPushConsumer.getAwaitTerminationMillisWhenShutdown());
throw new MQClientException("The consumer group[" + this.defaultMQPushConsumer.getConsumerGroup()
+ "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
null);
......@@ -829,8 +834,7 @@ public class DefaultMQPushConsumerImpl implements MQConsumerInner {
for (final Map.Entry<String, String> entry : sub.entrySet()) {
final String topic = entry.getKey();
final String subString = entry.getValue();
SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(this.defaultMQPushConsumer.getConsumerGroup(),
topic, subString);
SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(topic, subString);
this.rebalanceImpl.getSubscriptionInner().put(topic, subscriptionData);
}
}
......@@ -844,8 +848,7 @@ public class DefaultMQPushConsumerImpl implements MQConsumerInner {
break;
case CLUSTERING:
final String retryTopic = MixAll.getRetryTopic(this.defaultMQPushConsumer.getConsumerGroup());
SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(this.defaultMQPushConsumer.getConsumerGroup(),
retryTopic, SubscriptionData.SUB_ALL);
SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(retryTopic, SubscriptionData.SUB_ALL);
this.rebalanceImpl.getSubscriptionInner().put(retryTopic, subscriptionData);
break;
default:
......@@ -876,8 +879,7 @@ public class DefaultMQPushConsumerImpl implements MQConsumerInner {
public void subscribe(String topic, String subExpression) throws MQClientException {
try {
SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(this.defaultMQPushConsumer.getConsumerGroup(),
topic, subExpression);
SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(topic, subExpression);
this.rebalanceImpl.getSubscriptionInner().put(topic, subscriptionData);
if (this.mQClientFactory != null) {
this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
......@@ -889,8 +891,7 @@ public class DefaultMQPushConsumerImpl implements MQConsumerInner {
public void subscribe(String topic, String fullClassName, String filterClassSource) throws MQClientException {
try {
SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(this.defaultMQPushConsumer.getConsumerGroup(),
topic, "*");
SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(topic, "*");
subscriptionData.setSubString(fullClassName);
subscriptionData.setClassFilterMode(true);
subscriptionData.setFilterClassSource(filterClassSource);
......
client/src/main/java/org/apache/rocketmq/client/impl/consumer/ProcessQueue.java
浏览文件 @ 70295453
......@@ -44,11 +44,11 @@ public class ProcessQueue {
public final static long REBALANCE_LOCK_INTERVAL = Long.parseLong(System.getProperty("rocketmq.client.rebalance.lockInterval", "20000"));
private final static long PULL_MAX_IDLE_TIME = Long.parseLong(System.getProperty("rocketmq.client.pull.pullMaxIdleTime", "120000"));
private final InternalLogger log = ClientLogger.getLog();
private final ReadWriteLock lockTreeMap = new ReentrantReadWriteLock();
private final ReadWriteLock treeMapLock = new ReentrantReadWriteLock();
private final TreeMap<Long, MessageExt> msgTreeMap = new TreeMap<Long, MessageExt>();
private final AtomicLong msgCount = new AtomicLong();
private final AtomicLong msgSize = new AtomicLong();
private final Lock lockConsume = new ReentrantLock();
private final Lock consumeLock = new ReentrantLock();
/**
* A subset of msgTreeMap, will only be used when orderly consume
*/
......@@ -83,7 +83,7 @@ public class ProcessQueue {
for (int i = 0; i < loop; i++) {
MessageExt msg = null;
try {
this.lockTreeMap.readLock().lockInterruptibly();
this.treeMapLock.readLock().lockInterruptibly();
try {
if (!msgTreeMap.isEmpty() && System.currentTimeMillis() - Long.parseLong(MessageAccessor.getConsumeStartTimeStamp(msgTreeMap.firstEntry().getValue())) > pushConsumer.getConsumeTimeout() * 60 * 1000) {
msg = msgTreeMap.firstEntry().getValue();
......@@ -92,7 +92,7 @@ public class ProcessQueue {
break;
}
} finally {
this.lockTreeMap.readLock().unlock();
this.treeMapLock.readLock().unlock();
}
} catch (InterruptedException e) {
log.error("getExpiredMsg exception", e);
......@@ -103,7 +103,7 @@ public class ProcessQueue {
pushConsumer.sendMessageBack(msg, 3);
log.info("send expire msg back. topic={}, msgId={}, storeHost={}, queueId={}, queueOffset={}", msg.getTopic(), msg.getMsgId(), msg.getStoreHost(), msg.getQueueId(), msg.getQueueOffset());
try {
this.lockTreeMap.writeLock().lockInterruptibly();
this.treeMapLock.writeLock().lockInterruptibly();
try {
if (!msgTreeMap.isEmpty() && msg.getQueueOffset() == msgTreeMap.firstKey()) {
try {
......@@ -113,7 +113,7 @@ public class ProcessQueue {
}
}
} finally {
this.lockTreeMap.writeLock().unlock();
this.treeMapLock.writeLock().unlock();
}
} catch (InterruptedException e) {
log.error("getExpiredMsg exception", e);
......@@ -127,7 +127,7 @@ public class ProcessQueue {
public boolean putMessage(final List<MessageExt> msgs) {
boolean dispatchToConsume = false;
try {
this.lockTreeMap.writeLock().lockInterruptibly();
this.treeMapLock.writeLock().lockInterruptibly();
try {
int validMsgCnt = 0;
for (MessageExt msg : msgs) {
......@@ -156,7 +156,7 @@ public class ProcessQueue {
}
}
} finally {
this.lockTreeMap.writeLock().unlock();
this.treeMapLock.writeLock().unlock();
}
} catch (InterruptedException e) {
log.error("putMessage exception", e);
......@@ -167,13 +167,13 @@ public class ProcessQueue {
public long getMaxSpan() {
try {
this.lockTreeMap.readLock().lockInterruptibly();
this.treeMapLock.readLock().lockInterruptibly();
try {
if (!this.msgTreeMap.isEmpty()) {
return this.msgTreeMap.lastKey() - this.msgTreeMap.firstKey();
}
} finally {
this.lockTreeMap.readLock().unlock();
this.treeMapLock.readLock().unlock();
}
} catch (InterruptedException e) {
log.error("getMaxSpan exception", e);
......@@ -186,7 +186,7 @@ public class ProcessQueue {
long result = -1;
final long now = System.currentTimeMillis();
try {
this.lockTreeMap.writeLock().lockInterruptibly();
this.treeMapLock.writeLock().lockInterruptibly();
this.lastConsumeTimestamp = now;
try {
if (!msgTreeMap.isEmpty()) {
......@@ -206,7 +206,7 @@ public class ProcessQueue {
}
}
} finally {
this.lockTreeMap.writeLock().unlock();
this.treeMapLock.writeLock().unlock();
}
} catch (Throwable t) {
log.error("removeMessage exception", t);
......@@ -245,12 +245,12 @@ public class ProcessQueue {
public void rollback() {
try {
this.lockTreeMap.writeLock().lockInterruptibly();
this.treeMapLock.writeLock().lockInterruptibly();
try {
this.msgTreeMap.putAll(this.consumingMsgOrderlyTreeMap);
this.consumingMsgOrderlyTreeMap.clear();
} finally {
this.lockTreeMap.writeLock().unlock();
this.treeMapLock.writeLock().unlock();
}
} catch (InterruptedException e) {
log.error("rollback exception", e);
......@@ -259,7 +259,7 @@ public class ProcessQueue {
public long commit() {
try {
this.lockTreeMap.writeLock().lockInterruptibly();
this.treeMapLock.writeLock().lockInterruptibly();
try {
Long offset = this.consumingMsgOrderlyTreeMap.lastKey();
msgCount.addAndGet(0 - this.consumingMsgOrderlyTreeMap.size());
......@@ -271,7 +271,7 @@ public class ProcessQueue {
return offset + 1;
}
} finally {
this.lockTreeMap.writeLock().unlock();
this.treeMapLock.writeLock().unlock();
}
} catch (InterruptedException e) {
log.error("commit exception", e);
......@@ -280,16 +280,16 @@ public class ProcessQueue {
return -1;
}
public void makeMessageToCosumeAgain(List<MessageExt> msgs) {
public void makeMessageToConsumeAgain(List<MessageExt> msgs) {
try {
this.lockTreeMap.writeLock().lockInterruptibly();
this.treeMapLock.writeLock().lockInterruptibly();
try {
for (MessageExt msg : msgs) {
this.consumingMsgOrderlyTreeMap.remove(msg.getQueueOffset());
this.msgTreeMap.put(msg.getQueueOffset(), msg);
}
} finally {
this.lockTreeMap.writeLock().unlock();
this.treeMapLock.writeLock().unlock();
}
} catch (InterruptedException e) {
log.error("makeMessageToCosumeAgain exception", e);
......@@ -300,7 +300,7 @@ public class ProcessQueue {
List<MessageExt> result = new ArrayList<MessageExt>(batchSize);
final long now = System.currentTimeMillis();
try {
this.lockTreeMap.writeLock().lockInterruptibly();
this.treeMapLock.writeLock().lockInterruptibly();
this.lastConsumeTimestamp = now;
try {
if (!this.msgTreeMap.isEmpty()) {
......@@ -319,7 +319,7 @@ public class ProcessQueue {
consuming = false;
}
} finally {
this.lockTreeMap.writeLock().unlock();
this.treeMapLock.writeLock().unlock();
}
} catch (InterruptedException e) {
log.error("take Messages exception", e);
......@@ -330,11 +330,11 @@ public class ProcessQueue {
public boolean hasTempMessage() {
try {
this.lockTreeMap.readLock().lockInterruptibly();
this.treeMapLock.readLock().lockInterruptibly();
try {
return !this.msgTreeMap.isEmpty();
} finally {
this.lockTreeMap.readLock().unlock();
this.treeMapLock.readLock().unlock();
}
} catch (InterruptedException e) {
}
......@@ -344,7 +344,7 @@ public class ProcessQueue {
public void clear() {
try {
this.lockTreeMap.writeLock().lockInterruptibly();
this.treeMapLock.writeLock().lockInterruptibly();
try {
this.msgTreeMap.clear();
this.consumingMsgOrderlyTreeMap.clear();
......@@ -352,7 +352,7 @@ public class ProcessQueue {
this.msgSize.set(0);
this.queueOffsetMax = 0L;
} finally {
this.lockTreeMap.writeLock().unlock();
this.treeMapLock.writeLock().unlock();
}
} catch (InterruptedException e) {
log.error("rollback exception", e);
......@@ -367,8 +367,8 @@ public class ProcessQueue {
this.lastLockTimestamp = lastLockTimestamp;
}
public Lock getLockConsume() {
return lockConsume;
public Lock getConsumeLock() {
return consumeLock;
}
public long getLastPullTimestamp() {
......@@ -397,7 +397,7 @@ public class ProcessQueue {
public void fillProcessQueueInfo(final ProcessQueueInfo info) {
try {
this.lockTreeMap.readLock().lockInterruptibly();
this.treeMapLock.readLock().lockInterruptibly();
if (!this.msgTreeMap.isEmpty()) {
info.setCachedMsgMinOffset(this.msgTreeMap.firstKey());
......@@ -421,7 +421,7 @@ public class ProcessQueue {
info.setLastConsumeTimestamp(this.lastConsumeTimestamp);
} catch (Exception e) {
} finally {
this.lockTreeMap.readLock().unlock();
this.treeMapLock.readLock().unlock();
}
}
......
client/src/main/java/org/apache/rocketmq/client/impl/consumer/PullAPIWrapper.java
浏览文件 @ 70295453
......@@ -194,7 +194,7 @@ public class PullAPIWrapper {
String brokerAddr = findBrokerResult.getBrokerAddr();
if (PullSysFlag.hasClassFilterFlag(sysFlagInner)) {
brokerAddr = computPullFromWhichFilterServer(mq.getTopic(), brokerAddr);
brokerAddr = computePullFromWhichFilterServer(mq.getTopic(), brokerAddr);
}
PullResult pullResult = this.mQClientFactory.getMQClientAPIImpl().pullMessage(
......@@ -223,7 +223,7 @@ public class PullAPIWrapper {
return MixAll.MASTER_ID;
}
private String computPullFromWhichFilterServer(final String topic, final String brokerAddr)
private String computePullFromWhichFilterServer(final String topic, final String brokerAddr)
throws MQClientException {
ConcurrentMap<String, TopicRouteData> topicRouteTable = this.mQClientFactory.getTopicRouteTable();
if (topicRouteTable != null) {
......
client/src/main/java/org/apache/rocketmq/client/impl/consumer/PullRequest.java
浏览文件 @ 70295453
......@@ -23,14 +23,14 @@ public class PullRequest {
private MessageQueue messageQueue;
private ProcessQueue processQueue;
private long nextOffset;
private boolean lockedFirst = false;
private boolean previouslyLocked = false;
public boolean isLockedFirst() {
return lockedFirst;
public boolean isPreviouslyLocked() {
return previouslyLocked;
}
public void setLockedFirst(boolean lockedFirst) {
this.lockedFirst = lockedFirst;
public void setPreviouslyLocked(boolean previouslyLocked) {
this.previouslyLocked = previouslyLocked;
}
public String getConsumerGroup() {
......
client/src/main/java/org/apache/rocketmq/client/impl/consumer/RebalanceImpl.java
浏览文件 @ 70295453
......@@ -28,6 +28,7 @@ import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import org.apache.rocketmq.client.consumer.AllocateMessageQueueStrategy;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.client.impl.FindBrokerResult;
import org.apache.rocketmq.client.impl.factory.MQClientInstance;
import org.apache.rocketmq.client.log.ClientLogger;
......@@ -373,7 +374,15 @@ public abstract class RebalanceImpl {
this.removeDirtyOffset(mq);
ProcessQueue pq = new ProcessQueue();
long nextOffset = this.computePullFromWhere(mq);
long nextOffset = -1L;
try {
nextOffset = this.computePullFromWhereWithException(mq);
} catch (MQClientException e) {
log.info("doRebalance, {}, compute offset failed, {}", consumerGroup, mq);
continue;
}
if (nextOffset >= 0) {
ProcessQueue pre = this.processQueueTable.putIfAbsent(mq, pq);
if (pre != null) {
......@@ -408,8 +417,17 @@ public abstract class RebalanceImpl {
public abstract void removeDirtyOffset(final MessageQueue mq);
/**
* When the network is unstable, using this interface may return wrong offset.
* It is recommended to use computePullFromWhereWithException instead.
* @param mq
* @return offset
*/
@Deprecated
public abstract long computePullFromWhere(final MessageQueue mq);
public abstract long computePullFromWhereWithException(final MessageQueue mq) throws MQClientException;
public abstract void dispatchPullRequest(final List<PullRequest> pullRequestList);
public void removeProcessQueue(final MessageQueue mq) {
......
client/src/main/java/org/apache/rocketmq/client/impl/consumer/RebalanceLitePullImpl.java
浏览文件 @ 70295453
......@@ -74,8 +74,20 @@ public class RebalanceLitePullImpl extends RebalanceImpl {
this.litePullConsumerImpl.getOffsetStore().removeOffset(mq);
}
@Deprecated
@Override
public long computePullFromWhere(MessageQueue mq) {
long result = -1L;
try {
result = computePullFromWhereWithException(mq);
} catch (MQClientException e) {
log.warn("Compute consume offset exception, mq={}", mq);
}
return result;
}
@Override
public long computePullFromWhereWithException(MessageQueue mq) throws MQClientException {
ConsumeFromWhere consumeFromWhere = litePullConsumerImpl.getDefaultLitePullConsumer().getConsumeFromWhere();
long result = -1;
switch (consumeFromWhere) {
......@@ -118,7 +130,8 @@ public class RebalanceLitePullImpl extends RebalanceImpl {
try {
result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
} catch (MQClientException e) {
result = -1;
log.warn("Compute consume offset from last offset exception, mq={}, exception={}", mq, e);
throw e;
}
} else {
try {
......@@ -126,7 +139,8 @@ public class RebalanceLitePullImpl extends RebalanceImpl {
UtilAll.YYYYMMDDHHMMSS).getTime();
result = this.mQClientFactory.getMQAdminImpl().searchOffset(mq, timestamp);
} catch (MQClientException e) {
result = -1;
log.warn("Compute consume offset from last offset exception, mq={}, exception={}", mq, e);
throw e;
}
}
} else {
......
client/src/main/java/org/apache/rocketmq/client/impl/consumer/RebalancePullImpl.java
浏览文件 @ 70295453
......@@ -20,6 +20,7 @@ import java.util.List;
import java.util.Set;
import org.apache.rocketmq.client.consumer.AllocateMessageQueueStrategy;
import org.apache.rocketmq.client.consumer.MessageQueueListener;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.client.impl.factory.MQClientInstance;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.protocol.heartbeat.ConsumeType;
......@@ -68,11 +69,17 @@ public class RebalancePullImpl extends RebalanceImpl {
this.defaultMQPullConsumerImpl.getOffsetStore().removeOffset(mq);
}
@Deprecated
@Override
public long computePullFromWhere(MessageQueue mq) {
return 0;
}
@Override
public long computePullFromWhereWithException(MessageQueue mq) throws MQClientException {
return 0;
}
@Override
public void dispatchPullRequest(List<PullRequest> pullRequestList) {
}
......
client/src/main/java/org/apache/rocketmq/client/impl/consumer/RebalancePushImpl.java
浏览文件 @ 70295453
......@@ -88,11 +88,11 @@ public class RebalancePushImpl extends RebalanceImpl {
if (this.defaultMQPushConsumerImpl.isConsumeOrderly()
&& MessageModel.CLUSTERING.equals(this.defaultMQPushConsumerImpl.messageModel())) {
try {
if (pq.getLockConsume().tryLock(1000, TimeUnit.MILLISECONDS)) {
if (pq.getConsumeLock().tryLock(1000, TimeUnit.MILLISECONDS)) {
try {
return this.unlockDelay(mq, pq);
} finally {
pq.getLockConsume().unlock();
pq.getConsumeLock().unlock();
}
} else {
log.warn("[WRONG]mq is consuming, so can not unlock it, {}. maybe hanged for a while, {}",
......@@ -137,8 +137,20 @@ public class RebalancePushImpl extends RebalanceImpl {
this.defaultMQPushConsumerImpl.getOffsetStore().removeOffset(mq);
}
@Deprecated
@Override
public long computePullFromWhere(MessageQueue mq) {
long result = -1L;
try {
result = computePullFromWhereWithException(mq);
} catch (MQClientException e) {
log.warn("Compute consume offset exception, mq={}", mq);
}
return result;
}
@Override
public long computePullFromWhereWithException(MessageQueue mq) throws MQClientException {
long result = -1;
final ConsumeFromWhere consumeFromWhere = this.defaultMQPushConsumerImpl.getDefaultMQPushConsumer().getConsumeFromWhere();
final OffsetStore offsetStore = this.defaultMQPushConsumerImpl.getOffsetStore();
......@@ -159,7 +171,8 @@ public class RebalancePushImpl extends RebalanceImpl {
try {
result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
} catch (MQClientException e) {
result = -1;
log.warn("Compute consume offset from last offset exception, mq={}, exception={}", mq, e);
throw e;
}
}
} else {
......@@ -187,7 +200,8 @@ public class RebalancePushImpl extends RebalanceImpl {
try {
result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
} catch (MQClientException e) {
result = -1;
log.warn("Compute consume offset from last offset exception, mq={}, exception={}", mq, e);
throw e;
}
} else {
try {
......@@ -195,7 +209,8 @@ public class RebalancePushImpl extends RebalanceImpl {
UtilAll.YYYYMMDDHHMMSS).getTime();
result = this.mQClientFactory.getMQAdminImpl().searchOffset(mq, timestamp);
} catch (MQClientException e) {
result = -1;
log.warn("Compute consume offset from last offset exception, mq={}, exception={}", mq, e);
throw e;
}
}
} else {
......
client/src/main/java/org/apache/rocketmq/client/impl/factory/MQClientInstance.java
浏览文件 @ 70295453
......@@ -475,7 +475,7 @@ public class MQClientInstance {
this.lockHeartbeat.unlock();
}
} else {
log.warn("lock heartBeat, but failed.");
log.warn("lock heartBeat, but failed. [{}]", this.clientId);
}
}
......@@ -532,7 +532,7 @@ public class MQClientInstance {
final boolean producerEmpty = heartbeatData.getProducerDataSet().isEmpty();
final boolean consumerEmpty = heartbeatData.getConsumerDataSet().isEmpty();
if (producerEmpty && consumerEmpty) {
log.warn("sending heartbeat, but no consumer and no producer");
log.warn("sending heartbeat, but no consumer and no producer. [{}]", this.clientId);
return;
}
......@@ -668,10 +668,10 @@ public class MQClientInstance {
return true;
}
} else {
log.warn("updateTopicRouteInfoFromNameServer, getTopicRouteInfoFromNameServer return null, Topic: {}", topic);
log.warn("updateTopicRouteInfoFromNameServer, getTopicRouteInfoFromNameServer return null, Topic: {}. [{}]", topic, this.clientId);
}
} catch (MQClientException e) {
if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX) && !topic.equals(MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC)) {
if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
log.warn("updateTopicRouteInfoFromNameServer Exception", e);
}
} catch (RemotingException e) {
......@@ -681,7 +681,7 @@ public class MQClientInstance {
this.lockNamesrv.unlock();
}
} else {
log.warn("updateTopicRouteInfoFromNameServer tryLock timeout {}ms", LOCK_TIMEOUT_MILLIS);
log.warn("updateTopicRouteInfoFromNameServer tryLock timeout {}ms. [{}]", LOCK_TIMEOUT_MILLIS, this.clientId);
}
} catch (InterruptedException e) {
log.warn("updateTopicRouteInfoFromNameServer Exception", e);
......@@ -893,7 +893,7 @@ public class MQClientInstance {
this.lockHeartbeat.unlock();
}
} else {
log.warn("lock heartBeat, but failed.");
log.warn("lock heartBeat, but failed. [{}]", this.clientId);
}
} catch (InterruptedException e) {
log.warn("unregisterClientWithLock exception", e);
......@@ -1043,6 +1043,11 @@ public class MQClientInstance {
slave = brokerId != MixAll.MASTER_ID;
found = brokerAddr != null;
if (!found && slave) {
brokerAddr = map.get(brokerId + 1);
found = brokerAddr != null;
}
if (!found && !onlyThisBroker) {
Entry<Long, String> entry = map.entrySet().iterator().next();
brokerAddr = entry.getValue();
......@@ -1100,7 +1105,7 @@ public class MQClientInstance {
return null;
}
public void resetOffset(String topic, String group, Map<MessageQueue, Long> offsetTable) {
public synchronized void resetOffset(String topic, String group, Map<MessageQueue, Long> offsetTable) {
DefaultMQPushConsumerImpl consumer = null;
try {
MQConsumerInner impl = this.consumerTable.get(group);
......@@ -1209,6 +1214,9 @@ public class MQClientInstance {
public ConsumerRunningInfo consumerRunningInfo(final String consumerGroup) {
MQConsumerInner mqConsumerInner = this.consumerTable.get(consumerGroup);
if (mqConsumerInner == null) {
return null;
}
ConsumerRunningInfo consumerRunningInfo = mqConsumerInner.consumerRunningInfo();
......
client/src/main/java/org/apache/rocketmq/client/impl/producer/DefaultMQProducerImpl.java
浏览文件 @ 70295453
......@@ -44,6 +44,8 @@ import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.client.exception.RequestTimeoutException;
import org.apache.rocketmq.client.hook.CheckForbiddenContext;
import org.apache.rocketmq.client.hook.CheckForbiddenHook;
import org.apache.rocketmq.client.hook.EndTransactionContext;
import org.apache.rocketmq.client.hook.EndTransactionHook;
import org.apache.rocketmq.client.hook.SendMessageContext;
import org.apache.rocketmq.client.hook.SendMessageHook;
import org.apache.rocketmq.client.impl.CommunicationMode;
......@@ -101,6 +103,7 @@ public class DefaultMQProducerImpl implements MQProducerInner {
private final ConcurrentMap<String/* topic */, TopicPublishInfo> topicPublishInfoTable =
new ConcurrentHashMap<String, TopicPublishInfo>();
private final ArrayList<SendMessageHook> sendMessageHookList = new ArrayList<SendMessageHook>();
private final ArrayList<EndTransactionHook> endTransactionHookList = new ArrayList<EndTransactionHook>();
private final RPCHook rpcHook;
private final BlockingQueue<Runnable> asyncSenderThreadPoolQueue;
private final ExecutorService defaultAsyncSenderExecutor;
......@@ -171,6 +174,11 @@ public class DefaultMQProducerImpl implements MQProducerInner {
log.info("register sendMessage Hook, {}", hook.hookName());
}
public void registerEndTransactionHook(final EndTransactionHook hook) {
this.endTransactionHookList.add(hook);
log.info("register endTransaction Hook, {}", hook.hookName());
}
public void start() throws MQClientException {
this.start(true);
}
......@@ -386,6 +394,7 @@ public class DefaultMQProducerImpl implements MQProducerInner {
if (exception != null) {
remark = "checkLocalTransactionState Exception: " + RemotingHelper.exceptionSimpleDesc(exception);
}
doExecuteEndTransactionHook(msg, uniqueKey, brokerAddr, localTransactionState, true);
try {
DefaultMQProducerImpl.this.mQClientFactory.getMQClientAPIImpl().endTransactionOneway(brokerAddr, thisHeader, remark,
......@@ -421,6 +430,7 @@ public class DefaultMQProducerImpl implements MQProducerInner {
public void createTopic(String key, String newTopic, int queueNum, int topicSysFlag) throws MQClientException {
this.makeSureStateOK();
Validators.checkTopic(newTopic);
Validators.isSystemTopic(newTopic);
this.mQClientFactory.getMQAdminImpl().createTopic(key, newTopic, queueNum, topicSysFlag);
}
......@@ -966,6 +976,36 @@ public class DefaultMQProducerImpl implements MQProducerInner {
}
}
public boolean hasEndTransactionHook() {
return !this.endTransactionHookList.isEmpty();
}
public void executeEndTransactionHook(final EndTransactionContext context) {
if (!this.endTransactionHookList.isEmpty()) {
for (EndTransactionHook hook : this.endTransactionHookList) {
try {
hook.endTransaction(context);
} catch (Throwable e) {
log.warn("failed to executeEndTransactionHook", e);
}
}
}
}
public void doExecuteEndTransactionHook(Message msg, String msgId, String brokerAddr, LocalTransactionState state,
boolean fromTransactionCheck) {
if (hasEndTransactionHook()) {
EndTransactionContext context = new EndTransactionContext();
context.setProducerGroup(defaultMQProducer.getProducerGroup());
context.setBrokerAddr(brokerAddr);
context.setMessage(msg);
context.setMsgId(msgId);
context.setTransactionId(msg.getTransactionId());
context.setTransactionState(state);
context.setFromTransactionCheck(fromTransactionCheck);
executeEndTransactionHook(context);
}
}
/**
* DEFAULT ONEWAY -------------------------------------------------------
*/
......@@ -1114,7 +1154,7 @@ public class DefaultMQProducerImpl implements MQProducerInner {
mq = mQClientFactory.getClientConfig().queueWithNamespace(selector.select(messageQueueList, userMessage, arg));
} catch (Throwable e) {
throw new MQClientException("select message queue throwed exception.", e);
throw new MQClientException("select message queue threw exception.", e);
}
long costTime = System.currentTimeMillis() - beginStartTime;
......@@ -1265,7 +1305,7 @@ public class DefaultMQProducerImpl implements MQProducerInner {
}
try {
this.endTransaction(sendResult, localTransactionState, localException);
this.endTransaction(msg, sendResult, localTransactionState, localException);
} catch (Exception e) {
log.warn("local transaction execute " + localTransactionState + ", but end broker transaction failed", e);
}
......@@ -1289,6 +1329,7 @@ public class DefaultMQProducerImpl implements MQProducerInner {
}
public void endTransaction(
final Message msg,
final SendResult sendResult,
final LocalTransactionState localTransactionState,
final Throwable localException) throws RemotingException, MQBrokerException, InterruptedException, UnknownHostException {
......@@ -1317,6 +1358,7 @@ public class DefaultMQProducerImpl implements MQProducerInner {
break;
}
doExecuteEndTransactionHook(msg, sendResult.getMsgId(), brokerAddr, localTransactionState, false);
requestHeader.setProducerGroup(this.defaultMQProducer.getProducerGroup());
requestHeader.setTranStateTableOffset(sendResult.getQueueOffset());
requestHeader.setMsgId(sendResult.getMsgId());
......
client/src/main/java/org/apache/rocketmq/client/impl/producer/TopicPublishInfo.java
浏览文件 @ 70295453
......@@ -70,9 +70,9 @@ public class TopicPublishInfo {
if (lastBrokerName == null) {
return selectOneMessageQueue();
} else {
int index = this.sendWhichQueue.getAndIncrement();
for (int i = 0; i < this.messageQueueList.size(); i++) {
int pos = Math.abs(index++) % this.messageQueueList.size();
int index = this.sendWhichQueue.incrementAndGet();
int pos = Math.abs(index) % this.messageQueueList.size();
if (pos < 0)
pos = 0;
MessageQueue mq = this.messageQueueList.get(pos);
......@@ -85,7 +85,7 @@ public class TopicPublishInfo {
}
public MessageQueue selectOneMessageQueue() {
int index = this.sendWhichQueue.getAndIncrement();
int index = this.sendWhichQueue.incrementAndGet();
int pos = Math.abs(index) % this.messageQueueList.size();
if (pos < 0)
pos = 0;
......
client/src/main/java/org/apache/rocketmq/client/latency/LatencyFaultToleranceImpl.java
浏览文件 @ 70295453
......@@ -80,7 +80,7 @@ public class LatencyFaultToleranceImpl implements LatencyFaultTolerance<String>
if (half <= 0) {
return tmpList.get(0).getName();
} else {
final int i = this.whichItemWorst.getAndIncrement() % half;
final int i = this.whichItemWorst.incrementAndGet() % half;
return tmpList.get(i).getName();
}
}
......
client/src/main/java/org/apache/rocketmq/client/latency/MQFaultStrategy.java
浏览文件 @ 70295453
......@@ -58,16 +58,14 @@ public class MQFaultStrategy {
public MessageQueue selectOneMessageQueue(final TopicPublishInfo tpInfo, final String lastBrokerName) {
if (this.sendLatencyFaultEnable) {
try {
int index = tpInfo.getSendWhichQueue().getAndIncrement();
int index = tpInfo.getSendWhichQueue().incrementAndGet();
for (int i = 0; i < tpInfo.getMessageQueueList().size(); i++) {
int pos = Math.abs(index++) % tpInfo.getMessageQueueList().size();
if (pos < 0)
pos = 0;
MessageQueue mq = tpInfo.getMessageQueueList().get(pos);
if (latencyFaultTolerance.isAvailable(mq.getBrokerName())) {
if (null == lastBrokerName || mq.getBrokerName().equals(lastBrokerName))
return mq;
}
if (latencyFaultTolerance.isAvailable(mq.getBrokerName()))
return mq;
}
final String notBestBroker = latencyFaultTolerance.pickOneAtLeast();
......@@ -76,7 +74,7 @@ public class MQFaultStrategy {
final MessageQueue mq = tpInfo.selectOneMessageQueue();
if (notBestBroker != null) {
mq.setBrokerName(notBestBroker);
mq.setQueueId(tpInfo.getSendWhichQueue().getAndIncrement() % writeQueueNums);
mq.setQueueId(tpInfo.getSendWhichQueue().incrementAndGet() % writeQueueNums);
}
return mq;
} else {
......
client/src/main/java/org/apache/rocketmq/client/producer/DefaultMQProducer.java
浏览文件 @ 70295453
......@@ -29,6 +29,7 @@ import org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl;
import org.apache.rocketmq.client.log.ClientLogger;
import org.apache.rocketmq.client.trace.AsyncTraceDispatcher;
import org.apache.rocketmq.client.trace.TraceDispatcher;
import org.apache.rocketmq.client.trace.hook.EndTransactionTraceHookImpl;
import org.apache.rocketmq.client.trace.hook.SendMessageTraceHookImpl;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.message.Message;
......@@ -38,6 +39,7 @@ import org.apache.rocketmq.common.message.MessageDecoder;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.message.MessageId;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.remoting.RPCHook;
import org.apache.rocketmq.remoting.exception.RemotingException;
......@@ -74,7 +76,7 @@ public class DefaultMQProducer extends ClientConfig implements MQProducer {
/**
* Just for testing or demo program
*/
private String createTopicKey = MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC;
private String createTopicKey = TopicValidator.AUTO_CREATE_TOPIC_KEY_TOPIC;
/**
* Number of queues to create per default topic.
......@@ -166,6 +168,8 @@ public class DefaultMQProducer extends ClientConfig implements MQProducer {
traceDispatcher = dispatcher;
this.defaultMQProducerImpl.registerSendMessageHook(
new SendMessageTraceHookImpl(traceDispatcher));
this.defaultMQProducerImpl.registerEndTransactionHook(
new EndTransactionTraceHookImpl(traceDispatcher));
} catch (Throwable e) {
log.error("system mqtrace hook init failed ,maybe can't send msg trace data");
}
......@@ -251,6 +255,8 @@ public class DefaultMQProducer extends ClientConfig implements MQProducer {
traceDispatcher = dispatcher;
this.getDefaultMQProducerImpl().registerSendMessageHook(
new SendMessageTraceHookImpl(traceDispatcher));
this.defaultMQProducerImpl.registerEndTransactionHook(
new EndTransactionTraceHookImpl(traceDispatcher));
} catch (Throwable e) {
log.error("system mqtrace hook init failed ,maybe can't send msg trace data");
}
......@@ -916,6 +922,29 @@ public class DefaultMQProducer extends ClientConfig implements MQProducer {
return this.defaultMQProducerImpl.send(batch(msgs), messageQueue, timeout);
}
@Override
public void send(Collection<Message> msgs, SendCallback sendCallback) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
this.defaultMQProducerImpl.send(batch(msgs), sendCallback);
}
@Override
public void send(Collection<Message> msgs, SendCallback sendCallback,
long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
this.defaultMQProducerImpl.send(batch(msgs), sendCallback, timeout);
}
@Override
public void send(Collection<Message> msgs, MessageQueue mq,
SendCallback sendCallback) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
this.defaultMQProducerImpl.send(batch(msgs), queueWithNamespace(mq), sendCallback);
}
@Override
public void send(Collection<Message> msgs, MessageQueue mq,
SendCallback sendCallback, long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
this.defaultMQProducerImpl.send(batch(msgs), queueWithNamespace(mq), sendCallback, timeout);
}
/**
* Sets an Executor to be used for executing callback methods. If the Executor is not set, {@link
* NettyRemotingClient#publicExecutor} will be used.
......
client/src/main/java/org/apache/rocketmq/client/producer/MQProducer.java
浏览文件 @ 70295453
......@@ -99,7 +99,19 @@ public interface MQProducer extends MQAdmin {
SendResult send(final Collection<Message> msgs, final MessageQueue mq, final long timeout)
throws MQClientException, RemotingException, MQBrokerException, InterruptedException;
void send(final Collection<Message> msgs, final SendCallback sendCallback) throws MQClientException, RemotingException, MQBrokerException,
InterruptedException;
void send(final Collection<Message> msgs, final SendCallback sendCallback, final long timeout) throws MQClientException, RemotingException,
MQBrokerException, InterruptedException;
void send(final Collection<Message> msgs, final MessageQueue mq, final SendCallback sendCallback) throws MQClientException, RemotingException,
MQBrokerException, InterruptedException;
void send(final Collection<Message> msgs, final MessageQueue mq, final SendCallback sendCallback, final long timeout) throws MQClientException,
RemotingException, MQBrokerException, InterruptedException;
//for rpc
Message request(final Message msg, final long timeout) throws RequestTimeoutException, MQClientException,
RemotingException, MQBrokerException, InterruptedException;
......
client/src/main/java/org/apache/rocketmq/client/producer/TransactionMQProducer.java
浏览文件 @ 70295453
......@@ -51,6 +51,10 @@ public class TransactionMQProducer extends DefaultMQProducer {
super(namespace, producerGroup, rpcHook);
}
public TransactionMQProducer(final String namespace, final String producerGroup, RPCHook rpcHook, boolean enableMsgTrace, final String customizedTraceTopic) {
super(namespace, producerGroup, rpcHook, enableMsgTrace, customizedTraceTopic);
}
@Override
public void start() throws MQClientException {
this.defaultMQProducerImpl.initTransactionEnv();
......
client/src/main/java/org/apache/rocketmq/client/producer/selector/SelectMessageQueueByHash.java
浏览文件 @ 70295453
......@@ -25,12 +25,10 @@ public class SelectMessageQueueByHash implements MessageQueueSelector {
@Override
public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
int value = arg.hashCode();
int value = arg.hashCode() % mqs.size();
if (value < 0) {
value = Math.abs(value);
}
value = value % mqs.size();
return mqs.get(value);
}
}
client/src/main/java/org/apache/rocketmq/client/stat/ConsumerStatsManager.java
浏览文件 @ 70295453
......@@ -62,7 +62,7 @@ public class ConsumerStatsManager {
}
public void incPullRT(final String group, final String topic, final long rt) {
this.topicAndGroupPullRT.addValue(topic + "@" + group, (int) rt, 1);
this.topicAndGroupPullRT.addRTValue(topic + "@" + group, (int) rt, 1);
}
public void incPullTPS(final String group, final String topic, final long msgs) {
......@@ -70,7 +70,7 @@ public class ConsumerStatsManager {
}
public void incConsumeRT(final String group, final String topic, final long rt) {
this.topicAndGroupConsumeRT.addValue(topic + "@" + group, (int) rt, 1);
this.topicAndGroupConsumeRT.addRTValue(topic + "@" + group, (int) rt, 1);
}
public void incConsumeOKTPS(final String group, final String topic, final long msgs) {
......
client/src/main/java/org/apache/rocketmq/client/trace/AsyncTraceDispatcher.java
浏览文件 @ 70295453
......@@ -16,7 +16,6 @@
*/
package org.apache.rocketmq.client.trace;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.HashSet;
......@@ -28,7 +27,9 @@ import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;
import org.apache.commons.lang3.StringUtils;
import org.apache.rocketmq.client.AccessChannel;
import org.apache.rocketmq.client.common.ThreadLocalIndex;
......@@ -41,11 +42,11 @@ import org.apache.rocketmq.client.producer.DefaultMQProducer;
import org.apache.rocketmq.client.producer.MessageQueueSelector;
import org.apache.rocketmq.client.producer.SendCallback;
import org.apache.rocketmq.client.producer.SendResult;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.ThreadFactoryImpl;
import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.apache.rocketmq.logging.InternalLogger;
import org.apache.rocketmq.remoting.RPCHook;
......@@ -54,6 +55,7 @@ import static org.apache.rocketmq.client.trace.TraceConstants.TRACE_INSTANCE_NAM
public class AsyncTraceDispatcher implements TraceDispatcher {
private final static InternalLogger log = ClientLogger.getLog();
private final static AtomicInteger COUNTER = new AtomicInteger();
private final int queueSize;
private final int batchSize;
private final int maxMsgSize;
......@@ -62,7 +64,7 @@ public class AsyncTraceDispatcher implements TraceDispatcher {
// The last discard number of log
private AtomicLong discardCount;
private Thread worker;
private ArrayBlockingQueue<TraceContext> traceContextQueue;
private final ArrayBlockingQueue<TraceContext> traceContextQueue;
private ArrayBlockingQueue<Runnable> appenderQueue;
private volatile Thread shutDownHook;
private volatile boolean stopped = false;
......@@ -76,7 +78,7 @@ public class AsyncTraceDispatcher implements TraceDispatcher {
private String group;
private Type type;
public AsyncTraceDispatcher(String group, Type type,String traceTopicName, RPCHook rpcHook) {
public AsyncTraceDispatcher(String group, Type type, String traceTopicName, RPCHook rpcHook) {
// queueSize is greater than or equal to the n power of 2 of value
this.queueSize = 2048;
this.batchSize = 100;
......@@ -90,15 +92,15 @@ public class AsyncTraceDispatcher implements TraceDispatcher {
if (!UtilAll.isBlank(traceTopicName)) {
this.traceTopicName = traceTopicName;
} else {
this.traceTopicName = MixAll.RMQ_SYS_TRACE_TOPIC;
this.traceTopicName = TopicValidator.RMQ_SYS_TRACE_TOPIC;
}
this.traceExecutor = new ThreadPoolExecutor(//
10, //
20, //
1000 * 60, //
TimeUnit.MILLISECONDS, //
this.appenderQueue, //
new ThreadFactoryImpl("MQTraceSendThread_"));
10, //
20, //
1000 * 60, //
TimeUnit.MILLISECONDS, //
this.appenderQueue, //
new ThreadFactoryImpl("MQTraceSendThread_"));
traceProducer = getAndCreateTraceProducer(rpcHook);
}
......@@ -165,7 +167,7 @@ public class AsyncTraceDispatcher implements TraceDispatcher {
}
private String genGroupNameForTrace() {
return TraceConstants.GROUP_NAME_PREFIX + "-" + this.group + "-" + this.type ;
return TraceConstants.GROUP_NAME_PREFIX + "-" + this.group + "-" + this.type + "-" + COUNTER.incrementAndGet();
}
@Override
......@@ -178,10 +180,15 @@ public class AsyncTraceDispatcher implements TraceDispatcher {
}
@Override
public void flush() throws IOException {
public void flush() {
// The maximum waiting time for refresh,avoid being written all the time, resulting in failure to return.
long end = System.currentTimeMillis() + 500;
while (traceContextQueue.size() > 0 || appenderQueue.size() > 0 && System.currentTimeMillis() <= end) {
while (System.currentTimeMillis() <= end) {
synchronized (traceContextQueue) {
if (traceContextQueue.size() == 0 && appenderQueue.size() == 0) {
break;
}
}
try {
Thread.sleep(1);
} catch (InterruptedException e) {
......@@ -194,6 +201,7 @@ public class AsyncTraceDispatcher implements TraceDispatcher {
@Override
public void shutdown() {
this.stopped = true;
flush();
this.traceExecutor.shutdown();
if (isStarted.get()) {
traceProducer.shutdown();
......@@ -210,11 +218,7 @@ public class AsyncTraceDispatcher implements TraceDispatcher {
public void run() {
synchronized (this) {
if (!this.hasShutdown) {
try {
flush();
} catch (IOException e) {
log.error("system MQTrace hook shutdown failed ,maybe loss some trace data");
}
flush();
}
}
}
......@@ -240,25 +244,27 @@ public class AsyncTraceDispatcher implements TraceDispatcher {
public void run() {
while (!stopped) {
List<TraceContext> contexts = new ArrayList<TraceContext>(batchSize);
for (int i = 0; i < batchSize; i++) {
TraceContext context = null;
try {
//get trace data element from blocking Queue — traceContextQueue
context = traceContextQueue.poll(5, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
synchronized (traceContextQueue) {
for (int i = 0; i < batchSize; i++) {
TraceContext context = null;
try {
//get trace data element from blocking Queue - traceContextQueue
context = traceContextQueue.poll(5, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
}
if (context != null) {
contexts.add(context);
} else {
break;
}
}
if (context != null) {
contexts.add(context);
} else {
break;
if (contexts.size() > 0) {
AsyncAppenderRequest request = new AsyncAppenderRequest(contexts);
traceExecutor.submit(request);
} else if (AsyncTraceDispatcher.this.stopped) {
this.stopped = true;
}
}
if (contexts.size() > 0) {
AsyncAppenderRequest request = new AsyncAppenderRequest(contexts);
traceExecutor.submit(request);
} else if (AsyncTraceDispatcher.this.stopped) {
this.stopped = true;
}
}
}
......@@ -350,7 +356,7 @@ public class AsyncTraceDispatcher implements TraceDispatcher {
* Send message trace data
*
* @param keySet the keyset in this batch(including msgId in original message not offsetMsgId)
* @param data the message trace data in this batch
* @param data the message trace data in this batch
*/
private void sendTraceDataByMQ(Set<String> keySet, final String data, String dataTopic, String regionId) {
String traceTopic = traceTopicName;
......@@ -370,7 +376,7 @@ public class AsyncTraceDispatcher implements TraceDispatcher {
@Override
public void onException(Throwable e) {
log.info("send trace data ,the traceData is " + data);
log.error("send trace data failed, the traceData is {}", data, e);
}
};
if (traceBrokerSet.isEmpty()) {
......@@ -387,7 +393,7 @@ public class AsyncTraceDispatcher implements TraceDispatcher {
filterMqs.add(queue);
}
}
int index = sendWhichQueue.getAndIncrement();
int index = sendWhichQueue.incrementAndGet();
int pos = Math.abs(index) % filterMqs.size();
if (pos < 0) {
pos = 0;
......@@ -398,7 +404,7 @@ public class AsyncTraceDispatcher implements TraceDispatcher {
}
} catch (Exception e) {
log.info("send trace data,the traceData is" + data);
log.error("send trace data failed, the traceData is {}", data, e);
}
}
......
client/src/main/java/org/apache/rocketmq/client/trace/TraceBean.java
浏览文件 @ 70295453
......@@ -16,6 +16,7 @@
*/
package org.apache.rocketmq.client.trace;
import org.apache.rocketmq.client.producer.LocalTransactionState;
import org.apache.rocketmq.common.UtilAll;
import org.apache.rocketmq.common.message.MessageType;
......@@ -32,7 +33,9 @@ public class TraceBean {
private int retryTimes;
private int bodyLength;
private MessageType msgType;
private LocalTransactionState transactionState;
private String transactionId;
private boolean fromTransactionCheck;
public MessageType getMsgType() {
return msgType;
......@@ -141,4 +144,28 @@ public class TraceBean {
public void setBodyLength(int bodyLength) {
this.bodyLength = bodyLength;
}
public LocalTransactionState getTransactionState() {
return transactionState;
}
public void setTransactionState(LocalTransactionState transactionState) {
this.transactionState = transactionState;
}
public String getTransactionId() {
return transactionId;
}
public void setTransactionId(String transactionId) {
this.transactionId = transactionId;
}
public boolean isFromTransactionCheck() {
return fromTransactionCheck;
}
public void setFromTransactionCheck(boolean fromTransactionCheck) {
this.fromTransactionCheck = fromTransactionCheck;
}
}
client/src/main/java/org/apache/rocketmq/client/trace/TraceConstants.java
浏览文件 @ 70295453
......@@ -16,7 +16,7 @@
*/
package org.apache.rocketmq.client.trace;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.topic.TopicValidator;
public class TraceConstants {
......@@ -24,5 +24,21 @@ public class TraceConstants {
public static final char CONTENT_SPLITOR = (char) 1;
public static final char FIELD_SPLITOR = (char) 2;
public static final String TRACE_INSTANCE_NAME = "PID_CLIENT_INNER_TRACE_PRODUCER";
public static final String TRACE_TOPIC_PREFIX = MixAll.SYSTEM_TOPIC_PREFIX + "TRACE_DATA_";
public static final String TRACE_TOPIC_PREFIX = TopicValidator.SYSTEM_TOPIC_PREFIX + "TRACE_DATA_";
public static final String TO_PREFIX = "To_";
public static final String FROM_PREFIX = "From_";
public static final String END_TRANSACTION = "EndTransaction";
public static final String ROCKETMQ_SERVICE = "rocketmq";
public static final String ROCKETMQ_SUCCESS = "rocketmq.success";
public static final String ROCKETMQ_TAGS = "rocketmq.tags";
public static final String ROCKETMQ_KEYS = "rocketmq.keys";
public static final String ROCKETMQ_SOTRE_HOST = "rocketmq.store_host";
public static final String ROCKETMQ_BODY_LENGTH = "rocketmq.body_length";
public static final String ROCKETMQ_MSG_ID = "rocketmq.mgs_id";
public static final String ROCKETMQ_MSG_TYPE = "rocketmq.mgs_type";
public static final String ROCKETMQ_REGION_ID = "rocketmq.region_id";
public static final String ROCKETMQ_TRANSACTION_ID = "rocketmq.transaction_id";
public static final String ROCKETMQ_TRANSACTION_STATE = "rocketmq.transaction_state";
public static final String ROCKETMQ_IS_FROM_TRANSACTION_CHECK = "rocketmq.is_from_transaction_check";
public static final String ROCKETMQ_RETRY_TIMERS = "rocketmq.retry_times";
}
client/src/main/java/org/apache/rocketmq/client/trace/TraceDataEncoder.java
浏览文件 @ 70295453
......@@ -16,6 +16,7 @@
*/
package org.apache.rocketmq.client.trace;
import org.apache.rocketmq.client.producer.LocalTransactionState;
import org.apache.rocketmq.common.message.MessageType;
import java.util.ArrayList;
......@@ -62,6 +63,14 @@ public class TraceDataEncoder {
bean.setOffsetMsgId(line[12]);
pubContext.setSuccess(Boolean.parseBoolean(line[13]));
}
// compatible with the old version
if (line.length >= 15) {
bean.setOffsetMsgId(line[12]);
pubContext.setSuccess(Boolean.parseBoolean(line[13]));
bean.setClientHost(line[14]);
}
pubContext.setTraceBeans(new ArrayList<TraceBean>(1));
pubContext.getTraceBeans().add(bean);
resList.add(pubContext);
......@@ -94,7 +103,32 @@ public class TraceDataEncoder {
// add the context type
subAfterContext.setContextCode(Integer.parseInt(line[6]));
}
// compatible with the old version
if (line.length >= 9) {
subAfterContext.setTimeStamp(Long.parseLong(line[7]));
subAfterContext.setGroupName(line[8]);
}
resList.add(subAfterContext);
} else if (line[0].equals(TraceType.EndTransaction.name())) {
TraceContext endTransactionContext = new TraceContext();
endTransactionContext.setTraceType(TraceType.EndTransaction);
endTransactionContext.setTimeStamp(Long.parseLong(line[1]));
endTransactionContext.setRegionId(line[2]);
endTransactionContext.setGroupName(line[3]);
TraceBean bean = new TraceBean();
bean.setTopic(line[4]);
bean.setMsgId(line[5]);
bean.setTags(line[6]);
bean.setKeys(line[7]);
bean.setStoreHost(line[8]);
bean.setMsgType(MessageType.values()[Integer.parseInt(line[9])]);
bean.setTransactionId(line[10]);
bean.setTransactionState(LocalTransactionState.valueOf(line[11]));
bean.setFromTransactionCheck(Boolean.parseBoolean(line[12]));
endTransactionContext.setTraceBeans(new ArrayList<TraceBean>(1));
endTransactionContext.getTraceBeans().add(bean);
resList.add(endTransactionContext);
}
}
return resList;
......@@ -130,7 +164,7 @@ public class TraceDataEncoder {
.append(ctx.getCostTime()).append(TraceConstants.CONTENT_SPLITOR)//
.append(bean.getMsgType().ordinal()).append(TraceConstants.CONTENT_SPLITOR)//
.append(bean.getOffsetMsgId()).append(TraceConstants.CONTENT_SPLITOR)//
.append(ctx.isSuccess()).append(TraceConstants.FIELD_SPLITOR);
.append(ctx.isSuccess()).append(TraceConstants.FIELD_SPLITOR);//
}
break;
case SubBefore: {
......@@ -155,9 +189,27 @@ public class TraceDataEncoder {
.append(ctx.isSuccess()).append(TraceConstants.CONTENT_SPLITOR)//
.append(bean.getKeys()).append(TraceConstants.CONTENT_SPLITOR)//
.append(ctx.getContextCode()).append(TraceConstants.FIELD_SPLITOR);
}
}
break;
case EndTransaction: {
TraceBean bean = ctx.getTraceBeans().get(0);
sb.append(ctx.getTraceType()).append(TraceConstants.CONTENT_SPLITOR)//
.append(ctx.getTimeStamp()).append(TraceConstants.CONTENT_SPLITOR)//
.append(ctx.getRegionId()).append(TraceConstants.CONTENT_SPLITOR)//
.append(ctx.getGroupName()).append(TraceConstants.CONTENT_SPLITOR)//
.append(bean.getTopic()).append(TraceConstants.CONTENT_SPLITOR)//
.append(bean.getMsgId()).append(TraceConstants.CONTENT_SPLITOR)//
.append(bean.getTags()).append(TraceConstants.CONTENT_SPLITOR)//
.append(bean.getKeys()).append(TraceConstants.CONTENT_SPLITOR)//
.append(bean.getStoreHost()).append(TraceConstants.CONTENT_SPLITOR)//
.append(bean.getMsgType().ordinal()).append(TraceConstants.CONTENT_SPLITOR)//
.append(bean.getTransactionId()).append(TraceConstants.CONTENT_SPLITOR)//
.append(bean.getTransactionState().name()).append(TraceConstants.CONTENT_SPLITOR)//
.append(bean.isFromTransactionCheck()).append(TraceConstants.FIELD_SPLITOR);
}
break;
default:
}
transferBean.setTransData(sb.toString());
......
client/src/main/java/org/apache/rocketmq/client/trace/TraceType.java
浏览文件 @ 70295453
......@@ -20,4 +20,5 @@ public enum TraceType {
Pub,
SubBefore,
SubAfter,
EndTransaction,
}
client/src/main/java/org/apache/rocketmq/client/trace/TraceView.java 0 → 100644
浏览文件 @ 70295453
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.rocketmq.client.trace;
import java.util.ArrayList;
import java.util.List;
import org.apache.commons.codec.Charsets;
import org.apache.rocketmq.common.message.MessageExt;
public class TraceView {
private String msgId;
private String tags;
private String keys;
private String storeHost;
private String clientHost;
private int costTime;
private String msgType;
private String offSetMsgId;
private long timeStamp;
private long bornTime;
private String topic;
private String groupName;
private String status;
public static List<TraceView> decodeFromTraceTransData(String key, MessageExt messageExt) {
List<TraceView> messageTraceViewList = new ArrayList<TraceView>();
String messageBody = new String(messageExt.getBody(), Charsets.UTF_8);
if (messageBody == null || messageBody.length() <= 0) {
return messageTraceViewList;
}
List<TraceContext> traceContextList = TraceDataEncoder.decoderFromTraceDataString(messageBody);
for (TraceContext context : traceContextList) {
TraceView messageTraceView = new TraceView();
TraceBean traceBean = context.getTraceBeans().get(0);
if (!traceBean.getMsgId().equals(key)) {
continue;
}
messageTraceView.setCostTime(context.getCostTime());
messageTraceView.setGroupName(context.getGroupName());
if (context.isSuccess()) {
messageTraceView.setStatus("success");
} else {
messageTraceView.setStatus("failed");
}
messageTraceView.setKeys(traceBean.getKeys());
messageTraceView.setMsgId(traceBean.getMsgId());
messageTraceView.setTags(traceBean.getTags());
messageTraceView.setTopic(traceBean.getTopic());
messageTraceView.setMsgType(context.getTraceType().name());
messageTraceView.setOffSetMsgId(traceBean.getOffsetMsgId());
messageTraceView.setTimeStamp(context.getTimeStamp());
messageTraceView.setStoreHost(traceBean.getStoreHost());
messageTraceView.setClientHost(messageExt.getBornHostString());
messageTraceViewList.add(messageTraceView);
}
return messageTraceViewList;
}
public String getMsgId() {
return msgId;
}
public void setMsgId(String msgId) {
this.msgId = msgId;
}
public String getTags() {
return tags;
}
public void setTags(String tags) {
this.tags = tags;
}
public String getKeys() {
return keys;
}
public void setKeys(String keys) {
this.keys = keys;
}
public String getStoreHost() {
return storeHost;
}
public void setStoreHost(String storeHost) {
this.storeHost = storeHost;
}
public String getClientHost() {
return clientHost;
}
public void setClientHost(String clientHost) {
this.clientHost = clientHost;
}
public int getCostTime() {
return costTime;
}
public void setCostTime(int costTime) {
this.costTime = costTime;
}
public String getMsgType() {
return msgType;
}
public void setMsgType(String msgType) {
this.msgType = msgType;
}
public String getOffSetMsgId() {
return offSetMsgId;
}
public void setOffSetMsgId(String offSetMsgId) {
this.offSetMsgId = offSetMsgId;
}
public long getTimeStamp() {
return timeStamp;
}
public void setTimeStamp(long timeStamp) {
this.timeStamp = timeStamp;
}
public long getBornTime() {
return bornTime;
}
public void setBornTime(long bornTime) {
this.bornTime = bornTime;
}
public String getTopic() {
return topic;
}
public void setTopic(String topic) {
this.topic = topic;
}
public String getGroupName() {
return groupName;
}
public void setGroupName(String groupName) {
this.groupName = groupName;
}
public String getStatus() {
return status;
}
public void setStatus(String status) {
this.status = status;
}
}
\ No newline at end of file
client/src/main/java/org/apache/rocketmq/client/trace/hook/ConsumeMessageOpenTracingHookImpl.java 0 → 100644
浏览文件 @ 70295453
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.rocketmq.client.trace.hook;
import io.opentracing.Span;
import io.opentracing.SpanContext;
import io.opentracing.Tracer;
import io.opentracing.propagation.Format;
import io.opentracing.propagation.TextMapAdapter;
import io.opentracing.tag.Tags;
import org.apache.rocketmq.client.hook.ConsumeMessageContext;
import org.apache.rocketmq.client.hook.ConsumeMessageHook;
import org.apache.rocketmq.client.trace.TraceConstants;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.protocol.NamespaceUtil;
import java.util.ArrayList;
import java.util.List;
public class ConsumeMessageOpenTracingHookImpl implements ConsumeMessageHook {
private Tracer tracer;
public ConsumeMessageOpenTracingHookImpl(Tracer tracer) {
this.tracer = tracer;
}
@Override
public String hookName() {
return "ConsumeMessageOpenTracingHook";
}
@Override
public void consumeMessageBefore(ConsumeMessageContext context) {
if (context == null || context.getMsgList() == null || context.getMsgList().isEmpty()) {
return;
}
List<Span> spanList = new ArrayList<>();
for (MessageExt msg : context.getMsgList()) {
if (msg == null) {
continue;
}
Tracer.SpanBuilder spanBuilder = tracer
.buildSpan(TraceConstants.FROM_PREFIX + msg.getTopic())
.withTag(Tags.SPAN_KIND, Tags.SPAN_KIND_CONSUMER);
SpanContext spanContext = tracer.extract(Format.Builtin.TEXT_MAP, new TextMapAdapter(msg.getProperties()));
if (spanContext != null) {
spanBuilder.asChildOf(spanContext);
}
Span span = spanBuilder.start();
span.setTag(Tags.PEER_SERVICE, TraceConstants.ROCKETMQ_SERVICE);
span.setTag(Tags.MESSAGE_BUS_DESTINATION, NamespaceUtil.withoutNamespace(msg.getTopic()));
span.setTag(TraceConstants.ROCKETMQ_MSG_ID, msg.getMsgId());
span.setTag(TraceConstants.ROCKETMQ_TAGS, msg.getTags());
span.setTag(TraceConstants.ROCKETMQ_KEYS, msg.getKeys());
span.setTag(TraceConstants.ROCKETMQ_BODY_LENGTH, msg.getStoreSize());
span.setTag(TraceConstants.ROCKETMQ_RETRY_TIMERS, msg.getReconsumeTimes());
span.setTag(TraceConstants.ROCKETMQ_REGION_ID, msg.getProperty(MessageConst.PROPERTY_MSG_REGION));
spanList.add(span);
}
context.setMqTraceContext(spanList);
}
@Override
public void consumeMessageAfter(ConsumeMessageContext context) {
if (context == null || context.getMsgList() == null || context.getMsgList().isEmpty()) {
return;
}
List<Span> spanList = (List<Span>) context.getMqTraceContext();
if (spanList == null) {
return;
}
for (Span span : spanList) {
span.setTag(TraceConstants.ROCKETMQ_SUCCESS, context.isSuccess());
span.finish();
}
}
}
client/src/main/java/org/apache/rocketmq/client/trace/hook/ConsumeMessageTraceHookImpl.java
浏览文件 @ 70295453
......@@ -16,6 +16,7 @@
*/
package org.apache.rocketmq.client.trace.hook;
import java.util.Map;
import org.apache.rocketmq.client.consumer.listener.ConsumeReturnType;
import org.apache.rocketmq.client.hook.ConsumeMessageContext;
import org.apache.rocketmq.client.hook.ConsumeMessageHook;
......@@ -91,7 +92,7 @@ public class ConsumeMessageTraceHookImpl implements ConsumeMessageHook {
TraceContext subBeforeContext = (TraceContext) context.getMqTraceContext();
if (subBeforeContext.getTraceBeans() == null || subBeforeContext.getTraceBeans().size() < 1) {
// If subbefore bean is null ,skip it
// If subBefore bean is null ,skip it
return;
}
TraceContext subAfterContext = new TraceContext();
......@@ -101,13 +102,16 @@ public class ConsumeMessageTraceHookImpl implements ConsumeMessageHook {
subAfterContext.setRequestId(subBeforeContext.getRequestId());//
subAfterContext.setSuccess(context.isSuccess());//
// Caculate the cost time for processing messages
// Calculate the cost time for processing messages
int costTime = (int) ((System.currentTimeMillis() - subBeforeContext.getTimeStamp()) / context.getMsgList().size());
subAfterContext.setCostTime(costTime);//
subAfterContext.setTraceBeans(subBeforeContext.getTraceBeans());
String contextType = context.getProps().get(MixAll.CONSUME_CONTEXT_TYPE);
if (contextType != null) {
subAfterContext.setContextCode(ConsumeReturnType.valueOf(contextType).ordinal());
Map<String, String> props = context.getProps();
if (props != null) {
String contextType = props.get(MixAll.CONSUME_CONTEXT_TYPE);
if (contextType != null) {
subAfterContext.setContextCode(ConsumeReturnType.valueOf(contextType).ordinal());
}
}
localDispatcher.append(subAfterContext);
}
......
client/src/main/java/org/apache/rocketmq/client/trace/hook/EndTransactionOpenTracingHookImpl.java 0 → 100644
浏览文件 @ 70295453
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.rocketmq.client.trace.hook;
import io.opentracing.Span;
import io.opentracing.SpanContext;
import io.opentracing.Tracer;
import io.opentracing.propagation.Format;
import io.opentracing.propagation.TextMapAdapter;
import io.opentracing.tag.Tags;
import org.apache.rocketmq.client.hook.EndTransactionContext;
import org.apache.rocketmq.client.hook.EndTransactionHook;
import org.apache.rocketmq.client.trace.TraceConstants;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.common.message.MessageType;
public class EndTransactionOpenTracingHookImpl implements EndTransactionHook {
private Tracer tracer;
public EndTransactionOpenTracingHookImpl(Tracer tracer) {
this.tracer = tracer;
}
@Override
public String hookName() {
return "EndTransactionOpenTracingHook";
}
@Override
public void endTransaction(EndTransactionContext context) {
if (context == null) {
return;
}
Message msg = context.getMessage();
Tracer.SpanBuilder spanBuilder = tracer
.buildSpan(TraceConstants.END_TRANSACTION)
.withTag(Tags.SPAN_KIND, Tags.SPAN_KIND_PRODUCER);
SpanContext spanContext = tracer.extract(Format.Builtin.TEXT_MAP, new TextMapAdapter(msg.getProperties()));
if (spanContext != null) {
spanBuilder.asChildOf(spanContext);
}
Span span = spanBuilder.start();
span.setTag(Tags.PEER_SERVICE, TraceConstants.ROCKETMQ_SERVICE);
span.setTag(Tags.MESSAGE_BUS_DESTINATION, msg.getTopic());
span.setTag(TraceConstants.ROCKETMQ_TAGS, msg.getTags());
span.setTag(TraceConstants.ROCKETMQ_KEYS, msg.getKeys());
span.setTag(TraceConstants.ROCKETMQ_SOTRE_HOST, context.getBrokerAddr());
span.setTag(TraceConstants.ROCKETMQ_MSG_ID, context.getMsgId());
span.setTag(TraceConstants.ROCKETMQ_MSG_TYPE, MessageType.Trans_msg_Commit.name());
span.setTag(TraceConstants.ROCKETMQ_TRANSACTION_ID, context.getTransactionId());
span.setTag(TraceConstants.ROCKETMQ_TRANSACTION_STATE, context.getTransactionState().name());
span.setTag(TraceConstants.ROCKETMQ_IS_FROM_TRANSACTION_CHECK, context.isFromTransactionCheck());
span.finish();
}
}
client/src/main/java/org/apache/rocketmq/client/trace/hook/EndTransactionTraceHookImpl.java 0 → 100644
浏览文件 @ 70295453
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.rocketmq.client.trace.hook;
import org.apache.rocketmq.client.hook.EndTransactionContext;
import org.apache.rocketmq.client.hook.EndTransactionHook;
import org.apache.rocketmq.client.trace.AsyncTraceDispatcher;
import org.apache.rocketmq.client.trace.TraceBean;
import org.apache.rocketmq.client.trace.TraceContext;
import org.apache.rocketmq.client.trace.TraceDispatcher;
import org.apache.rocketmq.client.trace.TraceType;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.message.MessageType;
import org.apache.rocketmq.common.protocol.NamespaceUtil;
import java.util.ArrayList;
public class EndTransactionTraceHookImpl implements EndTransactionHook {
private TraceDispatcher localDispatcher;
public EndTransactionTraceHookImpl(TraceDispatcher localDispatcher) {
this.localDispatcher = localDispatcher;
}
@Override
public String hookName() {
return "EndTransactionTraceHook";
}
@Override
public void endTransaction(EndTransactionContext context) {
//if it is message trace data,then it doesn't recorded
if (context == null || context.getMessage().getTopic().startsWith(((AsyncTraceDispatcher) localDispatcher).getTraceTopicName())) {
return;
}
Message msg = context.getMessage();
//build the context content of TuxeTraceContext
TraceContext tuxeContext = new TraceContext();
tuxeContext.setTraceBeans(new ArrayList<TraceBean>(1));
tuxeContext.setTraceType(TraceType.EndTransaction);
tuxeContext.setGroupName(NamespaceUtil.withoutNamespace(context.getProducerGroup()));
//build the data bean object of message trace
TraceBean traceBean = new TraceBean();
traceBean.setTopic(NamespaceUtil.withoutNamespace(context.getMessage().getTopic()));
traceBean.setTags(context.getMessage().getTags());
traceBean.setKeys(context.getMessage().getKeys());
traceBean.setStoreHost(context.getBrokerAddr());
traceBean.setMsgType(MessageType.Trans_msg_Commit);
traceBean.setClientHost(((AsyncTraceDispatcher)localDispatcher).getHostProducer().getmQClientFactory().getClientId());
traceBean.setMsgId(context.getMsgId());
traceBean.setTransactionState(context.getTransactionState());
traceBean.setTransactionId(context.getTransactionId());
traceBean.setFromTransactionCheck(context.isFromTransactionCheck());
String regionId = msg.getProperty(MessageConst.PROPERTY_MSG_REGION);
if (regionId == null || regionId.isEmpty()) {
regionId = MixAll.DEFAULT_TRACE_REGION_ID;
}
tuxeContext.setRegionId(regionId);
tuxeContext.getTraceBeans().add(traceBean);
tuxeContext.setTimeStamp(System.currentTimeMillis());
localDispatcher.append(tuxeContext);
}
}
client/src/main/java/org/apache/rocketmq/client/trace/hook/SendMessageOpenTracingHookImpl.java 0 → 100644
浏览文件 @ 70295453
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.rocketmq.client.trace.hook;
import io.opentracing.Span;
import io.opentracing.SpanContext;
import io.opentracing.Tracer;
import io.opentracing.propagation.Format;
import io.opentracing.propagation.TextMapAdapter;
import io.opentracing.tag.Tags;
import org.apache.rocketmq.client.hook.SendMessageContext;
import org.apache.rocketmq.client.hook.SendMessageHook;
import org.apache.rocketmq.client.producer.SendStatus;
import org.apache.rocketmq.client.trace.TraceConstants;
import org.apache.rocketmq.common.message.Message;
public class SendMessageOpenTracingHookImpl implements SendMessageHook {
private Tracer tracer;
public SendMessageOpenTracingHookImpl(Tracer tracer) {
this.tracer = tracer;
}
@Override
public String hookName() {
return "SendMessageOpenTracingHook";
}
@Override
public void sendMessageBefore(SendMessageContext context) {
if (context == null) {
return;
}
Message msg = context.getMessage();
Tracer.SpanBuilder spanBuilder = tracer
.buildSpan(TraceConstants.TO_PREFIX + msg.getTopic())
.withTag(Tags.SPAN_KIND, Tags.SPAN_KIND_PRODUCER);
SpanContext spanContext = tracer.extract(Format.Builtin.TEXT_MAP, new TextMapAdapter(msg.getProperties()));
if (spanContext != null) {
spanBuilder.asChildOf(spanContext);
}
Span span = spanBuilder.start();
tracer.inject(span.context(), Format.Builtin.TEXT_MAP, new TextMapAdapter(msg.getProperties()));
span.setTag(Tags.PEER_SERVICE, TraceConstants.ROCKETMQ_SERVICE);
span.setTag(Tags.MESSAGE_BUS_DESTINATION, msg.getTopic());
span.setTag(TraceConstants.ROCKETMQ_TAGS, msg.getTags());
span.setTag(TraceConstants.ROCKETMQ_KEYS, msg.getKeys());
span.setTag(TraceConstants.ROCKETMQ_SOTRE_HOST, context.getBrokerAddr());
span.setTag(TraceConstants.ROCKETMQ_MSG_TYPE, context.getMsgType().name());
span.setTag(TraceConstants.ROCKETMQ_BODY_LENGTH, msg.getBody().length);
context.setMqTraceContext(span);
}
@Override
public void sendMessageAfter(SendMessageContext context) {
if (context == null || context.getMqTraceContext() == null) {
return;
}
if (context.getSendResult() == null) {
return;
}
if (context.getSendResult().getRegionId() == null) {
return;
}
Span span = (Span) context.getMqTraceContext();
span.setTag(TraceConstants.ROCKETMQ_SUCCESS, context.getSendResult().getSendStatus().equals(SendStatus.SEND_OK));
span.setTag(TraceConstants.ROCKETMQ_MSG_ID, context.getSendResult().getMsgId());
span.setTag(TraceConstants.ROCKETMQ_REGION_ID, context.getSendResult().getRegionId());
span.finish();
}
}
client/src/test/java/org/apache/rocketmq/client/ValidatorsTest.java
浏览文件 @ 70295453
......@@ -19,11 +19,12 @@ package org.apache.rocketmq.client;
import org.apache.commons.lang3.StringUtils;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.topic.TopicValidator;
import org.junit.Test;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Fail.failBecauseExceptionWasNotThrown;
import static org.junit.Assert.fail;
public class ValidatorsTest {
......@@ -47,17 +48,6 @@ public class ValidatorsTest {
}
}
@Test
public void testCheckTopic_UseDefaultTopic() {
String defaultTopic = MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC;
try {
Validators.checkTopic(defaultTopic);
failBecauseExceptionWasNotThrown(MQClientException.class);
} catch (MQClientException e) {
assertThat(e).hasMessageStartingWith(String.format("The topic[%s] is conflict with AUTO_CREATE_TOPIC_KEY_TOPIC.", defaultTopic));
}
}
@Test
public void testCheckTopic_BlankTopic() {
String blankTopic = "";
......@@ -80,4 +70,30 @@ public class ValidatorsTest {
assertThat(e).hasMessageStartingWith("The specified topic is longer than topic max length");
}
}
@Test
public void testIsSystemTopic() {
for (String topic : TopicValidator.getSystemTopicSet()) {
try {
Validators.isSystemTopic(topic);
fail("excepted MQClientException for system topic");
} catch (MQClientException e) {
assertThat(e.getResponseCode()).isEqualTo(-1);
assertThat(e.getErrorMessage()).isEqualTo(String.format("The topic[%s] is conflict with system topic.", topic));
}
}
}
@Test
public void testIsNotAllowedSendTopic() {
for (String topic : TopicValidator.getNotAllowedSendTopicSet()) {
try {
Validators.isNotAllowedSendTopic(topic);
fail("excepted MQClientException for blacklist topic");
} catch (MQClientException e) {
assertThat(e.getResponseCode()).isEqualTo(-1);
assertThat(e.getErrorMessage()).isEqualTo(String.format("Sending message to topic[%s] is forbidden.", topic));
}
}
}
}
client/src/test/java/org/apache/rocketmq/client/common/ThreadLocalIndexTest.java
浏览文件 @ 70295453
......@@ -22,11 +22,18 @@ import static org.assertj.core.api.Assertions.assertThat;
public class ThreadLocalIndexTest {
@Test
public void testGetAndIncrement() throws Exception {
public void testIncrementAndGet() throws Exception {
ThreadLocalIndex localIndex = new ThreadLocalIndex();
int initialVal = localIndex.getAndIncrement();
int initialVal = localIndex.incrementAndGet();
assertThat(localIndex.getAndIncrement()).isEqualTo(initialVal + 1);
assertThat(localIndex.incrementAndGet()).isEqualTo(initialVal + 1);
}
@Test
public void testIncrementAndGet2() throws Exception {
ThreadLocalIndex localIndex = new ThreadLocalIndex();
int initialVal = localIndex.incrementAndGet();
assertThat(initialVal >= 0);
}
}
\ No newline at end of file
client/src/test/java/org/apache/rocketmq/client/consumer/DefaultLitePullConsumerTest.java
浏览文件 @ 70295453
......@@ -24,6 +24,8 @@ import java.util.Collections;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.concurrent.ConcurrentMap;
import org.apache.commons.lang3.reflect.FieldUtils;
import org.apache.rocketmq.client.ClientConfig;
import org.apache.rocketmq.client.consumer.store.OffsetStore;
import org.apache.rocketmq.client.consumer.store.ReadOffsetType;
......@@ -48,16 +50,15 @@ import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.protocol.header.PullMessageRequestHeader;
import org.apache.rocketmq.common.protocol.heartbeat.MessageModel;
import org.apache.rocketmq.remoting.RPCHook;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mock;
import org.mockito.Spy;
import org.mockito.invocation.InvocationOnMock;
import org.mockito.junit.MockitoJUnitRunner;
import org.mockito.stubbing.Answer;
import org.powermock.api.mockito.PowerMockito;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Fail.failBecauseExceptionWasNotThrown;
......@@ -71,8 +72,7 @@ import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.spy;
import static org.mockito.Mockito.when;
@RunWith(PowerMockRunner.class)
@PrepareForTest(DefaultLitePullConsumerImpl.class)
@RunWith(MockitoJUnitRunner.class)
public class DefaultLitePullConsumerTest {
@Spy
private MQClientInstance mQClientFactory = MQClientManager.getInstance().getOrCreateMQClientInstance(new ClientConfig());
......@@ -81,6 +81,8 @@ public class DefaultLitePullConsumerTest {
private MQClientAPIImpl mQClientAPIImpl;
@Mock
private MQAdminImpl mQAdminImpl;
@Mock
private AssignedMessageQueue assignedMQ;
private RebalanceImpl rebalanceImpl;
private OffsetStore offsetStore;
......@@ -92,7 +94,10 @@ public class DefaultLitePullConsumerTest {
@Before
public void init() throws Exception {
PowerMockito.suppress(PowerMockito.method(DefaultLitePullConsumerImpl.class, "updateTopicSubscribeInfoWhenSubscriptionChanged"));
ConcurrentMap<String, MQClientInstance> factoryTable = (ConcurrentMap<String, MQClientInstance>) FieldUtils.readDeclaredField(MQClientManager.getInstance(), "factoryTable", true);
factoryTable.forEach((s, instance) -> instance.shutdown());
factoryTable.clear();
Field field = MQClientInstance.class.getDeclaredField("rebalanceService");
field.setAccessible(true);
RebalanceService rebalanceService = (RebalanceService) field.get(mQClientFactory);
......@@ -180,7 +185,9 @@ public class DefaultLitePullConsumerTest {
when(mQAdminImpl.minOffset(any(MessageQueue.class))).thenReturn(0L);
when(mQAdminImpl.maxOffset(any(MessageQueue.class))).thenReturn(500L);
MessageQueue messageQueue = createMessageQueue();
litePullConsumer.assign(Collections.singletonList(messageQueue));
List<MessageQueue> messageQueues = Collections.singletonList(messageQueue);
litePullConsumer.assign(messageQueues);
litePullConsumer.pause(messageQueues);
long offset = litePullConsumer.committed(messageQueue);
litePullConsumer.seek(messageQueue, offset);
Field field = DefaultLitePullConsumerImpl.class.getDeclaredField("assignedMessageQueue");
......@@ -196,7 +203,9 @@ public class DefaultLitePullConsumerTest {
when(mQAdminImpl.minOffset(any(MessageQueue.class))).thenReturn(0L);
when(mQAdminImpl.maxOffset(any(MessageQueue.class))).thenReturn(500L);
MessageQueue messageQueue = createMessageQueue();
litePullConsumer.assign(Collections.singletonList(messageQueue));
List<MessageQueue> messageQueues = Collections.singletonList(messageQueue);
litePullConsumer.assign(messageQueues);
litePullConsumer.pause(messageQueues);
litePullConsumer.seekToBegin(messageQueue);
Field field = DefaultLitePullConsumerImpl.class.getDeclaredField("assignedMessageQueue");
field.setAccessible(true);
......@@ -211,7 +220,9 @@ public class DefaultLitePullConsumerTest {
when(mQAdminImpl.minOffset(any(MessageQueue.class))).thenReturn(0L);
when(mQAdminImpl.maxOffset(any(MessageQueue.class))).thenReturn(500L);
MessageQueue messageQueue = createMessageQueue();
litePullConsumer.assign(Collections.singletonList(messageQueue));
List<MessageQueue> messageQueues = Collections.singletonList(messageQueue);
litePullConsumer.assign(messageQueues);
litePullConsumer.pause(messageQueues);
litePullConsumer.seekToEnd(messageQueue);
Field field = DefaultLitePullConsumerImpl.class.getDeclaredField("assignedMessageQueue");
field.setAccessible(true);
......@@ -226,7 +237,9 @@ public class DefaultLitePullConsumerTest {
when(mQAdminImpl.minOffset(any(MessageQueue.class))).thenReturn(0L);
when(mQAdminImpl.maxOffset(any(MessageQueue.class))).thenReturn(100L);
MessageQueue messageQueue = createMessageQueue();
litePullConsumer.assign(Collections.singletonList(messageQueue));
List<MessageQueue> messageQueues = Collections.singletonList(messageQueue);
litePullConsumer.assign(messageQueues);
litePullConsumer.pause(messageQueues);
try {
litePullConsumer.seek(messageQueue, -1);
failBecauseExceptionWasNotThrown(MQClientException.class);
......@@ -304,6 +317,53 @@ public class DefaultLitePullConsumerTest {
}
}
@Test
public void testPullTaskImpl_ProcessQueueNull() throws Exception {
DefaultLitePullConsumer litePullConsumer = createNotStartLitePullConsumer();
try {
MessageQueue messageQueue = createMessageQueue();
litePullConsumer.assign(Collections.singletonList(messageQueue));
Field field = DefaultLitePullConsumer.class.getDeclaredField("defaultLitePullConsumerImpl");
field.setAccessible(true);
// set ProcessQueue dropped = true
DefaultLitePullConsumerImpl localLitePullConsumerImpl = (DefaultLitePullConsumerImpl) field.get(litePullConsumer);
field = DefaultLitePullConsumerImpl.class.getDeclaredField("assignedMessageQueue");
field.setAccessible(true);
when(assignedMQ.isPaused(any(MessageQueue.class))).thenReturn(false);
when(assignedMQ.getProcessQueue(any(MessageQueue.class))).thenReturn(null);
litePullConsumer.start();
field.set(localLitePullConsumerImpl, assignedMQ);
List<MessageExt> result = litePullConsumer.poll(100);
assertThat(result.isEmpty()).isTrue();
} finally {
litePullConsumer.shutdown();
}
}
@Test
public void testPullTaskImpl_ProcessQueueDropped() throws Exception {
DefaultLitePullConsumer litePullConsumer = createNotStartLitePullConsumer();
try {
MessageQueue messageQueue = createMessageQueue();
litePullConsumer.assign(Collections.singletonList(messageQueue));
Field field = DefaultLitePullConsumer.class.getDeclaredField("defaultLitePullConsumerImpl");
field.setAccessible(true);
// set ProcessQueue dropped = true
DefaultLitePullConsumerImpl localLitePullConsumerImpl = (DefaultLitePullConsumerImpl) field.get(litePullConsumer);
field = DefaultLitePullConsumerImpl.class.getDeclaredField("assignedMessageQueue");
field.setAccessible(true);
AssignedMessageQueue assignedMessageQueue = (AssignedMessageQueue) field.get(localLitePullConsumerImpl);
assignedMessageQueue.getProcessQueue(messageQueue).setDropped(true);
litePullConsumer.start();
List<MessageExt> result = litePullConsumer.poll(100);
assertThat(result.isEmpty()).isTrue();
} finally {
litePullConsumer.shutdown();
}
}
@Test
public void testRegisterTopicMessageQueueChangeListener_Success() throws Exception {
flag = false;
......@@ -464,6 +524,30 @@ public class DefaultLitePullConsumerTest {
assertThat(offset).isEqualTo(100);
}
@Test
public void testConsumerAfterShutdown() throws Exception {
DefaultLitePullConsumer defaultLitePullConsumer = createSubscribeLitePullConsumer();
new AsyncConsumer().executeAsync(defaultLitePullConsumer);
Thread.sleep(100);
defaultLitePullConsumer.shutdown();
assertThat(defaultLitePullConsumer.isRunning()).isFalse();
}
static class AsyncConsumer {
public void executeAsync(final DefaultLitePullConsumer consumer) {
new Thread(new Runnable() {
@Override
public void run() {
while (consumer.isRunning()) {
List<MessageExt> poll = consumer.poll(2 * 1000);
}
}
}).start();
}
}
private void initDefaultLitePullConsumer(DefaultLitePullConsumer litePullConsumer) throws Exception {
Field field = DefaultLitePullConsumer.class.getDeclaredField("defaultLitePullConsumerImpl");
......@@ -500,9 +584,9 @@ public class DefaultLitePullConsumerTest {
when(mQClientFactory.getMQClientAPIImpl().pullMessage(anyString(), any(PullMessageRequestHeader.class),
anyLong(), any(CommunicationMode.class), nullable(PullCallback.class)))
.thenAnswer(new Answer<Object>() {
.thenAnswer(new Answer<PullResult>() {
@Override
public Object answer(InvocationOnMock mock) throws Throwable {
public PullResult answer(InvocationOnMock mock) throws Throwable {
PullMessageRequestHeader requestHeader = mock.getArgument(1);
MessageClientExt messageClientExt = new MessageClientExt();
messageClientExt.setTopic(topic);
......@@ -528,6 +612,7 @@ public class DefaultLitePullConsumerTest {
DefaultLitePullConsumer litePullConsumer = new DefaultLitePullConsumer(consumerGroup + System.currentTimeMillis());
litePullConsumer.setNamesrvAddr("127.0.0.1:9876");
litePullConsumer.subscribe(topic, "*");
suppressUpdateTopicRouteInfoFromNameServer(litePullConsumer);
litePullConsumer.start();
initDefaultLitePullConsumer(litePullConsumer);
return litePullConsumer;
......@@ -536,6 +621,7 @@ public class DefaultLitePullConsumerTest {
private DefaultLitePullConsumer createStartLitePullConsumer() throws Exception {
DefaultLitePullConsumer litePullConsumer = new DefaultLitePullConsumer(consumerGroup + System.currentTimeMillis());
litePullConsumer.setNamesrvAddr("127.0.0.1:9876");
suppressUpdateTopicRouteInfoFromNameServer(litePullConsumer);
litePullConsumer.start();
initDefaultLitePullConsumer(litePullConsumer);
return litePullConsumer;
......@@ -551,6 +637,7 @@ public class DefaultLitePullConsumerTest {
litePullConsumer.setNamesrvAddr("127.0.0.1:9876");
litePullConsumer.setMessageModel(MessageModel.BROADCASTING);
litePullConsumer.subscribe(topic, "*");
suppressUpdateTopicRouteInfoFromNameServer(litePullConsumer);
litePullConsumer.start();
initDefaultLitePullConsumer(litePullConsumer);
return litePullConsumer;
......@@ -572,4 +659,15 @@ public class DefaultLitePullConsumerTest {
}
return new PullResultExt(pullStatus, requestHeader.getQueueOffset() + messageExtList.size(), 123, 2048, messageExtList, 0, outputStream.toByteArray());
}
private static void suppressUpdateTopicRouteInfoFromNameServer(DefaultLitePullConsumer litePullConsumer) throws IllegalAccessException {
DefaultLitePullConsumerImpl defaultLitePullConsumerImpl = (DefaultLitePullConsumerImpl) FieldUtils.readDeclaredField(litePullConsumer, "defaultLitePullConsumerImpl", true);
if (litePullConsumer.getMessageModel() == MessageModel.CLUSTERING) {
litePullConsumer.changeInstanceNameToPID();
}
MQClientInstance mQClientFactory = spy(MQClientManager.getInstance().getOrCreateMQClientInstance(litePullConsumer, (RPCHook) FieldUtils.readDeclaredField(defaultLitePullConsumerImpl, "rpcHook", true)));
ConcurrentMap<String, MQClientInstance> factoryTable = (ConcurrentMap<String, MQClientInstance>) FieldUtils.readDeclaredField(MQClientManager.getInstance(), "factoryTable", true);
factoryTable.put(litePullConsumer.buildMQClientId(), mQClientFactory);
doReturn(false).when(mQClientFactory).updateTopicRouteInfoFromNameServer(anyString());
}
}
client/src/test/java/org/apache/rocketmq/client/consumer/DefaultMQPushConsumerTest.java
浏览文件 @ 70295453
......@@ -17,14 +17,17 @@
package org.apache.rocketmq.client.consumer;
import java.io.ByteArrayOutputStream;
import java.lang.reflect.Field;
import java.net.InetSocketAddress;
import java.util.Collections;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
import org.apache.commons.lang3.reflect.FieldUtils;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyContext;
......@@ -36,14 +39,15 @@ import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.client.impl.CommunicationMode;
import org.apache.rocketmq.client.impl.FindBrokerResult;
import org.apache.rocketmq.client.impl.MQClientAPIImpl;
import org.apache.rocketmq.client.impl.MQClientManager;
import org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService;
import org.apache.rocketmq.client.impl.consumer.ConsumeMessageOrderlyService;
import org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl;
import org.apache.rocketmq.client.impl.consumer.ProcessQueue;
import org.apache.rocketmq.client.impl.consumer.PullAPIWrapper;
import org.apache.rocketmq.client.impl.consumer.PullMessageService;
import org.apache.rocketmq.client.impl.consumer.PullRequest;
import org.apache.rocketmq.client.impl.consumer.PullResultExt;
import org.apache.rocketmq.client.impl.consumer.RebalanceImpl;
import org.apache.rocketmq.client.impl.consumer.RebalancePushImpl;
import org.apache.rocketmq.client.impl.factory.MQClientInstance;
import org.apache.rocketmq.common.message.MessageClientExt;
......@@ -51,6 +55,7 @@ import org.apache.rocketmq.common.message.MessageDecoder;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.protocol.header.PullMessageRequestHeader;
import org.apache.rocketmq.remoting.RPCHook;
import org.apache.rocketmq.remoting.exception.RemotingException;
import org.junit.After;
import org.junit.Assert;
......@@ -59,10 +64,8 @@ import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mock;
import org.mockito.invocation.InvocationOnMock;
import org.mockito.junit.MockitoJUnitRunner;
import org.mockito.stubbing.Answer;
import org.powermock.api.mockito.PowerMockito;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Fail.failBecauseExceptionWasNotThrown;
......@@ -72,11 +75,11 @@ import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.nullable;
import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.spy;
import static org.mockito.Mockito.when;
@RunWith(PowerMockRunner.class)
@PrepareForTest(DefaultMQPushConsumerImpl.class)
@RunWith(MockitoJUnitRunner.class)
public class DefaultMQPushConsumerTest {
private String consumerGroup;
private String topic = "FooBar";
......@@ -85,57 +88,21 @@ public class DefaultMQPushConsumerTest {
@Mock
private MQClientAPIImpl mQClientAPIImpl;
private PullAPIWrapper pullAPIWrapper;
private RebalanceImpl rebalanceImpl;
private RebalancePushImpl rebalancePushImpl;
private DefaultMQPushConsumer pushConsumer;
@Before
public void init() throws Exception {
consumerGroup = "FooBarGroup" + System.currentTimeMillis();
pushConsumer = new DefaultMQPushConsumer(consumerGroup);
pushConsumer.setNamesrvAddr("127.0.0.1:9876");
pushConsumer.setPullInterval(60 * 1000);
pushConsumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
return null;
}
});
DefaultMQPushConsumerImpl pushConsumerImpl = pushConsumer.getDefaultMQPushConsumerImpl();
PowerMockito.suppress(PowerMockito.method(DefaultMQPushConsumerImpl.class, "updateTopicSubscribeInfoWhenSubscriptionChanged"));
rebalancePushImpl = spy(new RebalancePushImpl(pushConsumer.getDefaultMQPushConsumerImpl()));
Field field = DefaultMQPushConsumerImpl.class.getDeclaredField("rebalanceImpl");
field.setAccessible(true);
field.set(pushConsumerImpl, rebalancePushImpl);
pushConsumer.subscribe(topic, "*");
pushConsumer.start();
mQClientFactory = spy(pushConsumerImpl.getmQClientFactory());
field = DefaultMQPushConsumerImpl.class.getDeclaredField("mQClientFactory");
field.setAccessible(true);
field.set(pushConsumerImpl, mQClientFactory);
ConcurrentMap<String, MQClientInstance> factoryTable = (ConcurrentMap<String, MQClientInstance>) FieldUtils.readDeclaredField(MQClientManager.getInstance(), "factoryTable", true);
factoryTable.forEach((s, instance) -> instance.shutdown());
factoryTable.clear();
field = MQClientInstance.class.getDeclaredField("mQClientAPIImpl");
field.setAccessible(true);
field.set(mQClientFactory, mQClientAPIImpl);
pullAPIWrapper = spy(new PullAPIWrapper(mQClientFactory, consumerGroup, false));
field = DefaultMQPushConsumerImpl.class.getDeclaredField("pullAPIWrapper");
field.setAccessible(true);
field.set(pushConsumerImpl, pullAPIWrapper);
pushConsumer.getDefaultMQPushConsumerImpl().getRebalanceImpl().setmQClientFactory(mQClientFactory);
mQClientFactory.registerConsumer(consumerGroup, pushConsumerImpl);
when(mQClientFactory.getMQClientAPIImpl().pullMessage(anyString(), any(PullMessageRequestHeader.class),
when(mQClientAPIImpl.pullMessage(anyString(), any(PullMessageRequestHeader.class),
anyLong(), any(CommunicationMode.class), nullable(PullCallback.class)))
.thenAnswer(new Answer<Object>() {
.thenAnswer(new Answer<PullResult>() {
@Override
public Object answer(InvocationOnMock mock) throws Throwable {
public PullResult answer(InvocationOnMock mock) throws Throwable {
PullMessageRequestHeader requestHeader = mock.getArgument(1);
MessageClientExt messageClientExt = new MessageClientExt();
messageClientExt.setTopic(topic);
......@@ -151,12 +118,43 @@ public class DefaultMQPushConsumerTest {
}
});
consumerGroup = "FooBarGroup" + System.currentTimeMillis();
pushConsumer = new DefaultMQPushConsumer(consumerGroup);
pushConsumer.setNamesrvAddr("127.0.0.1:9876");
pushConsumer.setPullInterval(60 * 1000);
pushConsumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
return null;
}
});
DefaultMQPushConsumerImpl pushConsumerImpl = pushConsumer.getDefaultMQPushConsumerImpl();
rebalancePushImpl = spy(new RebalancePushImpl(pushConsumer.getDefaultMQPushConsumerImpl()));
// suppress updateTopicRouteInfoFromNameServer
pushConsumer.changeInstanceNameToPID();
mQClientFactory = MQClientManager.getInstance().getOrCreateMQClientInstance(pushConsumer, (RPCHook) FieldUtils.readDeclaredField(pushConsumerImpl, "rpcHook", true));
FieldUtils.writeDeclaredField(mQClientFactory, "mQClientAPIImpl", mQClientAPIImpl, true);
mQClientFactory = spy(mQClientFactory);
factoryTable.put(pushConsumer.buildMQClientId(), mQClientFactory);
doReturn(false).when(mQClientFactory).updateTopicRouteInfoFromNameServer(anyString());
doReturn(new FindBrokerResult("127.0.0.1:10911", false)).when(mQClientFactory).findBrokerAddressInSubscribe(anyString(), anyLong(), anyBoolean());
doReturn(Collections.singletonList(mQClientFactory.getClientId())).when(mQClientFactory).findConsumerIdList(anyString(), anyString());
rebalanceImpl = spy(pushConsumerImpl.getRebalanceImpl());
doReturn(123L).when(rebalanceImpl).computePullFromWhereWithException(any(MessageQueue.class));
FieldUtils.writeDeclaredField(pushConsumerImpl, "rebalanceImpl", rebalanceImpl, true);
Set<MessageQueue> messageQueueSet = new HashSet<MessageQueue>();
messageQueueSet.add(createPullRequest().getMessageQueue());
pushConsumer.getDefaultMQPushConsumerImpl().updateTopicSubscribeInfo(topic, messageQueueSet);
doReturn(123L).when(rebalancePushImpl).computePullFromWhere(any(MessageQueue.class));
pushConsumerImpl.updateTopicSubscribeInfo(topic, messageQueueSet);
pushConsumer.subscribe(topic, "*");
pushConsumer.start();
}
@After
......@@ -172,12 +170,12 @@ public class DefaultMQPushConsumerTest {
@Test
public void testPullMessage_Success() throws InterruptedException, RemotingException, MQBrokerException {
final CountDownLatch countDownLatch = new CountDownLatch(1);
final MessageExt[] messageExts = new MessageExt[1];
final AtomicReference<MessageExt> messageAtomic = new AtomicReference<>();
pushConsumer.getDefaultMQPushConsumerImpl().setConsumeMessageService(new ConsumeMessageConcurrentlyService(pushConsumer.getDefaultMQPushConsumerImpl(), new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
messageExts[0] = msgs.get(0);
messageAtomic.set(msgs.get(0));
countDownLatch.countDown();
return null;
}
......@@ -185,20 +183,22 @@ public class DefaultMQPushConsumerTest {
PullMessageService pullMessageService = mQClientFactory.getPullMessageService();
pullMessageService.executePullRequestImmediately(createPullRequest());
countDownLatch.await();
assertThat(messageExts[0].getTopic()).isEqualTo(topic);
assertThat(messageExts[0].getBody()).isEqualTo(new byte[] {'a'});
countDownLatch.await(10, TimeUnit.SECONDS);
MessageExt msg = messageAtomic.get();
assertThat(msg).isNotNull();
assertThat(msg.getTopic()).isEqualTo(topic);
assertThat(msg.getBody()).isEqualTo(new byte[] {'a'});
}
@Test
public void testPullMessage_SuccessWithOrderlyService() throws Exception {
final CountDownLatch countDownLatch = new CountDownLatch(1);
final MessageExt[] messageExts = new MessageExt[1];
final AtomicReference<MessageExt> messageAtomic = new AtomicReference<>();
MessageListenerOrderly listenerOrderly = new MessageListenerOrderly() {
@Override
public ConsumeOrderlyStatus consumeMessage(List<MessageExt> msgs, ConsumeOrderlyContext context) {
messageExts[0] = msgs.get(0);
messageAtomic.set(msgs.get(0));
countDownLatch.countDown();
return null;
}
......@@ -211,8 +211,10 @@ public class DefaultMQPushConsumerTest {
pullMessageService.executePullRequestLater(createPullRequest(), 100);
countDownLatch.await(10, TimeUnit.SECONDS);
assertThat(messageExts[0].getTopic()).isEqualTo(topic);
assertThat(messageExts[0].getBody()).isEqualTo(new byte[] {'a'});
MessageExt msg = messageAtomic.get();
assertThat(msg).isNotNull();
assertThat(msg.getTopic()).isEqualTo(topic);
assertThat(msg.getBody()).isEqualTo(new byte[] {'a'});
}
@Test
......@@ -256,6 +258,34 @@ public class DefaultMQPushConsumerTest {
}
}
@Test
public void testGracefulShutdown() throws InterruptedException, RemotingException, MQBrokerException, MQClientException {
final CountDownLatch countDownLatch = new CountDownLatch(1);
pushConsumer.setAwaitTerminationMillisWhenShutdown(2000);
final AtomicBoolean messageConsumedFlag = new AtomicBoolean(false);
pushConsumer.getDefaultMQPushConsumerImpl().setConsumeMessageService(new ConsumeMessageConcurrentlyService(pushConsumer.getDefaultMQPushConsumerImpl(), new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
countDownLatch.countDown();
try {
Thread.sleep(1000);
messageConsumedFlag.set(true);
} catch (InterruptedException e) {
}
return null;
}
}));
PullMessageService pullMessageService = mQClientFactory.getPullMessageService();
pullMessageService.executePullRequestImmediately(createPullRequest());
assertThat(countDownLatch.await(10, TimeUnit.SECONDS)).isTrue();
pushConsumer.shutdown();
assertThat(messageConsumedFlag.get()).isTrue();
}
private DefaultMQPushConsumer createPushConsumer() {
DefaultMQPushConsumer pushConsumer = new DefaultMQPushConsumer(consumerGroup);
pushConsumer.registerMessageListener(new MessageListenerConcurrently() {
......@@ -294,4 +324,21 @@ public class DefaultMQPushConsumerTest {
}
return new PullResultExt(pullStatus, requestHeader.getQueueOffset() + messageExtList.size(), 123, 2048, messageExtList, 0, outputStream.toByteArray());
}
@Test
public void testPullMessage_ExceptionOccursWhenComputePullFromWhere() throws MQClientException {
final CountDownLatch countDownLatch = new CountDownLatch(1);
final MessageExt[] messageExts = new MessageExt[1];
pushConsumer.getDefaultMQPushConsumerImpl().setConsumeMessageService(
new ConsumeMessageConcurrentlyService(pushConsumer.getDefaultMQPushConsumerImpl(),
(msgs, context) -> {
messageExts[0] = msgs.get(0);
return null;
}));
pushConsumer.getDefaultMQPushConsumerImpl().setConsumeOrderly(true);
PullMessageService pullMessageService = mQClientFactory.getPullMessageService();
pullMessageService.executePullRequestImmediately(createPullRequest());
assertThat(messageExts[0]).isNull();
}
}
client/src/test/java/org/apache/rocketmq/client/consumer/store/RemoteBrokerOffsetStoreTest.java
浏览文件 @ 70295453
......@@ -84,7 +84,7 @@ public class RemoteBrokerOffsetStoreTest {
offsetStore.updateOffset(messageQueue, 1024, false);
doThrow(new MQBrokerException(-1, ""))
doThrow(new MQBrokerException(-1, "", null))
.when(mqClientAPI).queryConsumerOffset(anyString(), any(QueryConsumerOffsetRequestHeader.class), anyLong());
assertThat(offsetStore.readOffset(messageQueue, ReadOffsetType.READ_FROM_STORE)).isEqualTo(-1);
......
client/src/test/java/org/apache/rocketmq/client/producer/selector/SelectMessageQueueByHashTest.java
浏览文件 @ 70295453
......@@ -44,6 +44,14 @@ public class SelectMessageQueueByHashTest {
String anotherOrderId = "234";
MessageQueue selected = selector.select(messageQueues, message, orderId);
assertThat(selector.select(messageQueues, message, anotherOrderId)).isNotEqualTo(selected);
//No exception is thrown while order Id hashcode is Integer.MIN
anotherOrderId = "polygenelubricants";
selector.select(messageQueues, message, anotherOrderId);
anotherOrderId = "GydZG_";
selector.select(messageQueues, message, anotherOrderId);
anotherOrderId = "DESIGNING WORKHOUSES";
selector.select(messageQueues, message, anotherOrderId);
}
}
\ No newline at end of file
client/src/test/resources/log4j2.xml 0 → 100644
浏览文件 @ 70295453
此差异已折叠。
common/pom.xml
浏览文件 @ 70295453
此差异已折叠。
distribution/LICENSE-BIN
浏览文件 @ 70295453
此差异已折叠。
distribution/NOTICE-BIN
浏览文件 @ 70295453
此差异已折叠。
distribution/benchmark/batchproducer.sh 0 → 100644
浏览文件 @ 70295453
此差异已折叠。
distribution/benchmark/runclass.sh
浏览文件 @ 70295453
此差异已折叠。
distribution/benchmark/shutdown.sh 0 → 100644
浏览文件 @ 70295453
此差异已折叠。
distribution/benchmark/tproducer.sh
浏览文件 @ 70295453
此差异已折叠。
distribution/bin/mqadmin
浏览文件 @ 70295453
此差异已折叠。
distribution/bin/runbroker.cmd
浏览文件 @ 70295453
此差异已折叠。
distribution/bin/runbroker.sh
浏览文件 @ 70295453
此差异已折叠。
distribution/bin/runserver.cmd
浏览文件 @ 70295453
此差异已折叠。
distribution/bin/runserver.sh
浏览文件 @ 70295453
此差异已折叠。
distribution/bin/tools.sh
浏览文件 @ 70295453
此差异已折叠。
distribution/pom.xml
浏览文件 @ 70295453
此差异已折叠。
distribution/release.xml
浏览文件 @ 70295453
此差异已折叠。
docs/cn/Configuration_System.md 0 → 100644
浏览文件 @ 70295453
此差异已折叠。
docs/cn/RocketMQ_Example.md
浏览文件 @ 70295453
此差异已折叠。
docs/cn/architecture.md
浏览文件 @ 70295453
此差异已折叠。
docs/cn/best_practice.md
浏览文件 @ 70295453
此差异已折叠。
docs/cn/concept.md
浏览文件 @ 70295453
此差异已折叠。
docs/cn/design.md
浏览文件 @ 70295453
此差异已折叠。
docs/cn/msg_trace/user_guide.md
浏览文件 @ 70295453
此差异已折叠。
docs/cn/operation.md
浏览文件 @ 70295453
此差异已折叠。
docs/cn/rpc_request.md
浏览文件 @ 70295453
此差异已折叠。
docs/en/Design_LoadBlancing.md
浏览文件 @ 70295453
此差异已折叠。
docs/en/Example_Filter.md
浏览文件 @ 70295453
此差异已折叠。
docs/en/Example_Transaction.md
浏览文件 @ 70295453
此差异已折叠。
example/pom.xml
浏览文件 @ 70295453
此差异已折叠。
filter/pom.xml
浏览文件 @ 70295453
此差异已折叠。
logappender/pom.xml
浏览文件 @ 70295453
此差异已折叠。
logging/pom.xml
浏览文件 @ 70295453
此差异已折叠。
namesrv/pom.xml
浏览文件 @ 70295453
此差异已折叠。
openmessaging/pom.xml
浏览文件 @ 70295453
此差异已折叠。
pom.xml
浏览文件 @ 70295453
此差异已折叠。
remoting/pom.xml
浏览文件 @ 70295453
此差异已折叠。
srvutil/pom.xml
浏览文件 @ 70295453
此差异已折叠。
store/pom.xml
浏览文件 @ 70295453
此差异已折叠。
test/pom.xml
浏览文件 @ 70295453
此差异已折叠。
test/src/test/resources/log4j.xml
浏览文件 @ 70295453
此差异已折叠。
tools/pom.xml
浏览文件 @ 70295453
此差异已折叠。
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或