- 03 9月, 2020 7 次提交
-
-
由 Kimura Takesi 提交于
Co-authored-by: NTakesi Kimura <takeshki@yahoo-corp.jp>
-
由 sijia-w 提交于
-
由 feynmanlin 提交于
Fixes #7799 ### Motivation Pulsar metrics providing wrong information ### Modifications add a reset for counter
-
由 Hao Zhang 提交于
Link [https://github.com/apache/pulsar/issues/7758](https://github.com/apache/pulsar/issues/7758) and master issue [https://github.com/apache/pulsar/issues/2688](https://github.com/apache/pulsar/issues/2688) ### Motivation Support set/get/remove maxProducers on a topic level.
-
由 feynmanlin 提交于
### Motivation PR #7818 #7802 supports topic-level policies. But the pulsar admin cli java doc is not supported accordingly.
-
由 Aloys 提交于
-
由 Boyang Jerry Peng 提交于
Co-authored-by: NJerry Peng <jerryp@splunk.com>
-
- 02 9月, 2020 11 次提交
-
-
由 Yuri Mizushima 提交于
[pulsar-broker] Stop to dispatch when skip message temporally since Key_Shared consumer stuck on delivery (#7553) ### Motivation In some case of Key_Shared consumer, messages ordering was broken. Here is how to reproduce(I think it is one of case to reproduce this issue). 1. Connect Consumer1 to Key_Shared subscription `sub` and stop to receive - receiverQueueSize: 500 2. Connect Producer and publish 500 messages with key `(i % 10)` 3. Connect Consumer2 to same subscription and start to receive - receiverQueueSize: 1 - since https://github.com/apache/pulsar/pull/7106 , Consumer2 can't receive (expected) 4. Producer publish more 500 messages with same key generation algorithm 5. After that, Consumer1 start to receive 6. Check Consumer2 message ordering - sometimes message ordering was broken in same key Consumer1: ``` Connected: Tue Jul 14 09:36:39 JST 2020 [pulsar-client-io-1-1] WARN com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider [pulsar-timer-4-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - [persistent://public/default/key-shared-test] [sub0] [820f0] Prefetched messages: 499 --- Consume throughput received: 0.02 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 0.00 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0 Received: my-message-0 PublishTime: 1594687006203 Date: Tue Jul 14 09:37:46 JST 2020 Received: my-message-1 PublishTime: 1594687006243 Date: Tue Jul 14 09:37:46 JST 2020 Received: my-message-2 PublishTime: 1594687006247 Date: Tue Jul 14 09:37:46 JST 2020 ... Received: my-message-498 PublishTime: 1594687008727 Date: Tue Jul 14 09:37:46 JST 2020 Received: my-message-499 PublishTime: 1594687008731 Date: Tue Jul 14 09:37:46 JST 2020 Received: my-message-500 PublishTime: 1594687038742 Date: Tue Jul 14 09:37:46 JST 2020 ... Received: my-message-990 PublishTime: 1594687040094 Date: Tue Jul 14 09:37:46 JST 2020 Received: my-message-994 PublishTime: 1594687040103 Date: Tue Jul 14 09:37:46 JST 2020 Received: my-message-995 PublishTime: 1594687040105 Date: Tue Jul 14 09:37:46 JST 2020 Received: my-message-997 PublishTime: 1594687040113 Date: Tue Jul 14 09:37:46 JST 2020 ``` Consumer2: ``` Connected: Tue Jul 14 09:37:03 JST 2020 [pulsar-client-io-1-1] WARN com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider Received: my-message-501 MessageId: 4:1501:-1 PublishTime: 1594687038753 Date: Tue Jul 14 09:37:46 JST 2020 Received: my-message-502 MessageId: 4:1502:-1 PublishTime: 1594687038755 Date: Tue Jul 14 09:37:46 JST 2020 Received: my-message-503 MessageId: 4:1503:-1 PublishTime: 1594687038759 Date: Tue Jul 14 09:37:46 JST 2020 Received: my-message-506 MessageId: 4:1506:-1 PublishTime: 1594687038785 Date: Tue Jul 14 09:37:46 JST 2020 Received: my-message-508 MessageId: 4:1508:-1 PublishTime: 1594687038812 Date: Tue Jul 14 09:37:46 JST 2020 Received: my-message-901 MessageId: 4:1901:-1 PublishTime: 1594687039871 Date: Tue Jul 14 09:37:46 JST 2020 Received: my-message-509 MessageId: 4:1509:-1 PublishTime: 1594687038815 Date: Tue Jul 14 09:37:46 JST 2020 ordering was broken, key: 1 oldNum: 901 newNum: 511 Received: my-message-511 MessageId: 4:1511:-1 PublishTime: 1594687038826 Date: Tue Jul 14 09:37:46 JST 2020 Received: my-message-512 MessageId: 4:1512:-1 PublishTime: 1594687038830 Date: Tue Jul 14 09:37:46 JST 2020 ... ``` I think this issue is caused by https://github.com/apache/pulsar/pull/7105. Here is an example. 1. dispatch messages 2. Consumer2 was stuck and `totalMessagesSent=0` - Consumer2 availablePermits was 0 3. skip redeliver messages temporally - Consumer2 availablePermits was back to 1 4. dispatch new messages - new message was dispatched to Consumer2 5. back to redeliver messages 4. dispatch messages - ordering was broken ### Modifications Stop to dispatch when skip message temporally since Key_Shared consumer stuck on delivery.
-
由 ran 提交于
Fixes https://github.com/streamnative/pulsar/issues/1312 # Motivation Currently, the transaction abort on partitions operation is not getting through. ### Modifications Make the transaction abort on partitions operation get through.
-
由 Marvin Cai 提交于
[Issue 7864][docs]Update document for how to pick active consumer for failover subscription. (#7917) Update document to specify how active consumer is picked for partitioned and non-partitoned topic for failover subscription mode. The change was push in 2.4.2 but document has not been updated. Fixes #7864
-
由 Yong Zhang 提交于
**Motivation** After #7813, the python client supports the oauth2 authentication as well. Add an example for it.
-
由 Frank J Kelly 提交于
[Issue 7933][pulsar-broker-common] Add some logging to improve Authentication debugging and Fix typos in code "occured" -> "occurred" (#7934) Fixes #7933 ### Motivation Newbie trying to make his first contribution to this projection :-) ### Modifications Added some logging to Authentication Service to help debugging when there are more than one AuthN provider. When I did that I noticed a typo ('occured' should have two r's) so I decided to try to fix all of them
-
由 hangc0276 提交于
Modifications Support set publish rate on topic level. Support get publish rate on topic level. Support remove publish rate on topic level.
-
由 冉小龙 提交于
Signed-off-by: Nxiaolong.ran <rxl@apache.org>
-
由 Jennifer Huang 提交于
-
由 Jennifer Huang 提交于
* Add Pulsar Summit CFP blog * update * Update 2020-08-25-pulsar-summit-asia-2020-cfp.md
-
由 冉小龙 提交于
Signed-off-by: Nxiaolong.ran <rxl@apache.org> ### Modifications Fix the time of release 2.6.1 to `08/21`
-
由 Tboy 提交于
-
- 01 9月, 2020 3 次提交
-
-
由 HuanliMeng 提交于
Fixes #7481 Motivation Default bookkeeper conf that ships with apache pulsar indicates that the unit of openLedgerRereplicationGracePeriod is in seconds, but official bookkeeper doc(and code) mentions that the unit is in milliseconds. Pulsar should configure default as 30000 instead of 30 and bookkeeper.conf should mention that the unit of this period is milliseconds instead of seconds. Modifications Update the bookkeep.config file Update the following Pulsar docs for releases 2.5.2, 2.6.0, 2.6.1, and master Reference > Pulsar Config > BookKeeper
-
由 lipenghui 提交于
-
由 ran 提交于
Motivation Currently, we have to know the batchSize to generate BatchMessageAcker. If we could get the batch index ack bitSet from Broker we could generate the BatchMessageAcker by the bitSet, this is useful for consuming transaction messages, we don't need to change the protocol to get the total message number of one transaction. Modifications Add a new static method to generate the BatchMessageAcker by BitSet.
-
- 31 8月, 2020 2 次提交
-
-
由 Tboy 提交于
-
由 ran 提交于
### Motivation The transaction message consuming logic needs to be optimized. One transaction commit marker in the topic partition consists of many Entries in the TransactionBuffer, so one transaction could be seen as one batch. ### Modifications 1. When sending messages to consumers use the `batchIndex` in the MessageIdData to present the startBatchIndex of the Entry. 2. Calculate the transaction messages number. 3. When the first reading one transaction init it's batchDeletedIndexes in the ManagedCursorImpl.
-
- 30 8月, 2020 3 次提交
-
-
由 hangc0276 提交于
Fix #7826 ### Motivation Support compaction threshold on topic level. Based on the system topic function. ### Modifications Support set compaction threshold on topic level. Support get compaction threshold on topic level. Support remove compaction threshold on topic level.
-
由 Boyang Jerry Peng 提交于
Co-authored-by: NJerry Peng <jerryp@splunk.com>
-
由 Boyang Jerry Peng 提交于
Co-authored-by: NJerry Peng <jerryp@splunk.com>
-
- 29 8月, 2020 2 次提交
-
-
由 Jennifer Huang 提交于
-
由 sijia-w 提交于
Update AWS deployment in reference-configuration.
-
- 28 8月, 2020 9 次提交
-
-
由 jianyun 提交于
### Motivation *Compatible with flink 1.11 need to use java8 date api in pulsar's primitive schemas.* ### Modifications *Add Instant, LocalDate, LocalTime, LocalDateTime to pulsar's primitive schemas* ### Verifying this change Add Instant, LocalDate, LocalTime, LocalDateTime types to the Schema type test
-
由 Yong Zhang 提交于
**Motivation** Migrate the build website job to github actions.
-
由 HuanliMeng 提交于
### Motivation PR (https://github.com/apache/pulsar/pull/7821) supports set/get/remove the deduplication policy at the topic level. ### Modifications Update the doc based on code updates. The following docs are updated: - Messaging: message deduplication - Manage message deduplication - pulsar admin cli reference
-
由 Jennifer Huang 提交于
Modifications Sync the Pulsar project description with the latest version on home page.
-
由 hangc0276 提交于
### Motivation Support message dispatch rate on topic level. Based on the system topic function. ### Modifications Support set message dispatch rate on topic level. Support get message dispatch rate on topic level. Support remove message dispatch rate on topic level.
-
由 Boyang Jerry Peng 提交于
Co-authored-by: NJerry Peng <jerryp@splunk.com>
-
由 Sanjeev Kulkarni 提交于
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>
-
由 Masahiro Sakamoto 提交于
### Motivation A deadlock occurred on our Pulsar 2.4.2 broker server. The cause is the following two threads: ``` "prometheus-stats-36-1" #410 prio=5 os_prio=0 tid=0x00007f4b70019800 nid=0x30ca waiting for monitor entry [0x00007f4bbe3b7000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.pulsar.broker.service.persistent.PersistentSubscription.getNumberOfEntriesDelayed(PersistentSubscription.java:1013) - waiting to lock <0x00007f913d098dd0> (a org.apache.pulsar.broker.service.persistent.PersistentSubscription) at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$getTopicStats$8(NamespaceStatsAggregator.java:129) at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator$$Lambda$523/2109257042.accept(Unknown Source) at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:385) at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.getTopicStats(NamespaceStatsAggregator.java:122) at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$null$0(NamespaceStatsAggregator.java:64) at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator$$Lambda$521/1017174654.accept(Unknown Source) at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:385) at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$null$1(NamespaceStatsAggregator.java:63) at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator$$Lambda$520/1098830264.accept(Unknown Source) at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:385) at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$2(NamespaceStatsAggregator.java:62) at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator$$Lambda$316/212211274.accept(Unknown Source) at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:385) at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.generate(NamespaceStatsAggregator.java:59) at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator.generate(PrometheusMetricsGenerator.java:73) at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsServlet.lambda$doGet$0(PrometheusMetricsServlet.java:70) at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsServlet$$Lambda$315/1221766138.run(Unknown Source) at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) "ForkJoinPool.commonPool-worker-104" #953 daemon prio=5 os_prio=0 tid=0x00007f4dc8030800 nid=0x3b87 waiting on condition [0x00007f48f6ce1000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00007f913d08b5c8> (a org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section) at java.util.concurrent.locks.StampedLock.acquireRead(StampedLock.java:1215) at java.util.concurrent.locks.StampedLock.readLock(StampedLock.java:428) at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:377) at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159) at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.values(ConcurrentOpenHashMap.java:174) at org.apache.pulsar.broker.service.persistent.PersistentTopic.getNumberOfConsumers(PersistentTopic.java:1227) at org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers.isConsumersExceededOnTopic(PersistentDispatcherMultipleConsumers.java:178) at org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers.addConsumer(PersistentDispatcherMultipleConsumers.java:144) - locked <0x00007f91120dc258> (a org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers) at org.apache.pulsar.broker.service.persistent.PersistentSubscription.addConsumer(PersistentSubscription.java:238) - locked <0x00007f913d098dd0> (a org.apache.pulsar.broker.service.persistent.PersistentSubscription) at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$subscribe$11(PersistentTopic.java:590) at org.apache.pulsar.broker.service.persistent.PersistentTopic$$Lambda$451/1414070467.accept(Unknown Source) at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683) at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010) at org.apache.pulsar.broker.service.persistent.PersistentTopic.subscribe(PersistentTopic.java:584) at org.apache.pulsar.broker.service.ServerCnx.lambda$null$10(ServerCnx.java:699) at org.apache.pulsar.broker.service.ServerCnx$$Lambda$408/1168861154.apply(Unknown Source) at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995) at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137) at org.apache.pulsar.broker.service.ServerCnx.lambda$null$13(ServerCnx.java:682) at org.apache.pulsar.broker.service.ServerCnx$$Lambda$406/1351396211.apply(Unknown Source) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575) at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:943) at java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:457) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163) ``` `prometheus-stats-36-1` was trying to lock `PersistentSubscription` (and `PersistentDispatcherMultipleConsumers`) after locking `PersistentTopic#subscriptions`, an instance of `ConcurrentOpenHashMap`. https://github.com/apache/pulsar/blob/v2.4.2/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java#L122-L129 On the other hand, `ForkJoinPool.commonPool-worker-104` was trying to lock these instances in reverse order. https://github.com/apache/pulsar/blob/v2.4.2/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentSubscription.java#L176-L237 https://github.com/apache/pulsar/blob/v2.4.2/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java#L127-L144 https://github.com/apache/pulsar/blob/v2.4.2/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java#L159-L175 https://github.com/apache/pulsar/blob/v2.4.2/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1212-L1214 `PersistentSubscription#getNumberOfEntriesDelayed()` is no longer used in the master code, but it seems that this deadlock has not yet been resolved. https://github.com/apache/pulsar/blob/e06e8726847584700d9e4fc98fd56a495eb05a23/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1528-L1529 https://github.com/apache/pulsar/blob/17ae233a5d0fa364048b7c30ec90b8f7291d0d07/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentSubscription.java#L967-L1000 https://github.com/apache/pulsar/blob/6e7d1a83c3c2737610f01cb372f61e2b830a62f7/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java#L809 ### Modifications Moved the `isConsumersExceededOnTopic()` method to check the number of connected consumers when adding a consumer to the topic from the `Dispatcher` classes to the `AbstractTopic` class. Avoid the deadlock mentioned above by calling `PersistentTopic#getNumberOfConsumers()` before locking the `PersistentSubscription` instance.
-
由 Masahiro Sakamoto 提交于
Fixes #7554 ### Motivation As mentioned in #7554, the class of `individualDeletedMessages` is different between Pulsar 2.3.2 (and earlier) and 2.4.0 (and later). This causes some of ranges contained in `individualDeletedMessages` to be lost when the version of Pulsar is upgraded, and a large number of messages that have already been acked can be redelivered to consumers. Also, even if the Pulsar version is 2.4.0 or later, the same phenomenon occurs when the value of `managedLedgerUnackedRangesOpenCacheSetEnabled` is switched from false to true. ### Modifications If the list of individually deleted message ranges loaded from ZK contains ranges that span different ledgers, split the ranges by ledger ID and store them in `individualDeletedMessages`. As a result, information about deleted message ranges is never lost and messages that have already been acked will not be redelivered.
-
- 27 8月, 2020 1 次提交
-
-
由 ran 提交于
### Motivation Currently, the Pulsar SQL CI test is blocked. ### Modifications 1. Build a new docker image before CI test to make sure the latest changes are covered. 2. Remove the config `distributed-joins-enabled` in the `$PULSAR_HOME/conf/presto/config.properties`, this config was discarded in presto version `332`. 3. Add the config `-Dpresto-temporarily-allow-java8=true` in `$PULSAR_HOME/conf/presto/jvm.config`, this config is needed if use Java 8 environment. 4. The ClientBuilder will be loaded by two different ClassLoaders, in the Pulsar SQL CI test the ClientBuilder will be created by unexpected ClassLoader, we need to specify the ClassLoader for it.
-
- 26 8月, 2020 1 次提交
-
-
由 冉小龙 提交于
-
- 25 8月, 2020 1 次提交
-
-
由 Caleb Epstein 提交于
Fixes #7787 ### Motivation Throw std::exception instead of `const char*` ### Modifications Should be self explanatory. ### Verifying this change This change should be covered by existing tests, such as BatchMessageTest. Not sure about the Auth test which catches `...` however.
-