- 28 8月, 2019 25 次提交
-
-
由 Xiaobing Fang 提交于
fix #4755 Retention policy is used in Ledger GC and Topic GC. Default retention policy is not uploaded to zookeeper, but it is getted from zookeeper when it is used. So, if zookeeper doesn't have retention policy, we should load it from default config file. In Ledger GC configuration, it's OK. https://github.com/apache/pulsar/blob/075f28b71c8fd9259ce8e136dbb81c0629c3f271/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L739-L742 But in Topic GC, do nothing. (cherry picked from commit 1dcac8cc)
-
由 Boyang Jerry Peng 提交于
### Modifications When submitting a function via REST interface, if user doesn't provide the function package or function runtime as part of function config, the function will get submitted and deployed to run but the typeClassName for the source will not be set causing a weird error in the function instance code (cherry picked from commit e1547e12)
-
由 Boyang Jerry Peng 提交于
### Motivation A message doesn't get automatically ACKed if there is the output topic is not set (cherry picked from commit 19b883b7)
-
由 tuteng 提交于
### Motivation Currently our JDBC Sink not support deletion and update events. Support for delete and update events. ### Modifications Support for delete and update events. Add some document for JDBC Sink. https://github.com/apache/pulsar/issues/4073 ### Verifying this change local Unit Test pass. Integration test pass
-
由 Rui Fu 提交于
### Motivation `pulsar-function-go/conf` package apply `instance-conf-path` with default value `HOME_PATH+github.com/apache/pulsar/pulsar-function-go/conf/conf.yaml`, once function deployed, the running node may not have the yaml conf file exist, then go function will panic with `not found conf file` error. This PR changed the logic of config parsing, parse `confContent` first, then parse `confFilePath` if `confContent` empty. (cherry picked from commit 8a3b3af6)
-
由 Matteo Merli 提交于
### Motivation After the changes in #3118, there has a been a sharp increase of memory utilization for the UnackedMessageTracker due to the time buckets being created. This is especially true when the acktimeout is set to a larger value (eg: 1h) where 3600 time-buckets are being created. This lead to use 20MB per partition even when no message is tracked. Allowing to configure the tick time so that application can tune it based on needs. Additionally, fixed the logic that keeps creating hash maps and throwing them away at each tick time iteration, since that creates a lot of garbage and doesn't take care of the fact that the hash maps are expanding based on the required capacity (so next time they are already of the "right" size). On a final note: the current default of 1sec seems very wasteful. Something like 10s should be more appropriate as default. (cherry picked from commit f13af487)
-
由 Boyang Jerry Peng 提交于
### Motivation Currently, if the kubernetes namespace set to deploy functions in is different than the one in which brokers/workers reside, get status and stats doesn't work because the url for instances does not specify the namespace. (cherry picked from commit 8c3445ad)
-
由 Boyang Jerry Peng 提交于
(cherry picked from commit 26fc4efe)
-
由 Rui Fu 提交于
### Motivation Currently golang function needs to be compiled before deploy to pulsar, so the executable permission is required when function package is downloaded to local node from bookkeeper. This PR is intent to make golang function package executable after download from bookkeeper, to make sure the function is ok to run. (cherry picked from commit bf58c8eb)
-
由 Jia Zhai 提交于
**Motivation** when create partitioned topic, there is a check that `numPartitions > 1`, if numPartitions==1, it will fail. Some user may want to create partitioned topic with only 1 topic at the start time, and during using it, could update to more topics later. **Modification** change check of `numPartitions > 1` to `numPartitions > 0` expect all existing ut passed. (cherry picked from commit 5162393c)
-
由 Yuto Furuta 提交于
(cherry picked from commit 9605aede)
-
由 massakam 提交于
### Motivation If trying to build master C++ code with the `USE_LOG4CXX` flag turned ON, compilation errors occur: ``` /tmp/pulsar/pulsar-client-cpp/lib/Log4cxxLogger.cc: In static member function ‘static log4cxx::LevelPtr pulsar::Log4CxxLogger::getLevel(pulsar::Logger::Level)’: /tmp/pulsar/pulsar-client-cpp/lib/Log4cxxLogger.cc:53:18: error: ‘DEBUG’ was not declared in this scope case DEBUG: ^ /tmp/pulsar/pulsar-client-cpp/lib/Log4cxxLogger.cc:55:18: error: ‘INFO’ was not declared in this scope case INFO: ^ /tmp/pulsar/pulsar-client-cpp/lib/Log4cxxLogger.cc:57:18: error: ‘WARN’ was not declared in this scope case WARN: ^ /tmp/pulsar/pulsar-client-cpp/lib/Log4cxxLogger.cc:59:18: error: ‘ERROR’ was not declared in this scope case ERROR: ^ /tmp/pulsar/pulsar-client-cpp/lib/Log4cxxLogger.cc:52:16: error: enumeration value ‘LEVEL_DEBUG’ not handled in switch [-Werror=switch] switch (level) { ^ /tmp/pulsar/pulsar-client-cpp/lib/Log4cxxLogger.cc:52:16: error: enumeration value ‘LEVEL_INFO’ not handled in switch [-Werror=switch] /tmp/pulsar/pulsar-client-cpp/lib/Log4cxxLogger.cc:52:16: error: enumeration value ‘LEVEL_WARN’ not handled in switch [-Werror=switch] /tmp/pulsar/pulsar-client-cpp/lib/Log4cxxLogger.cc:52:16: error: enumeration value ‘LEVEL_ERROR’ not handled in switch [-Werror=switch] cc1plus: some warnings being treated as errors make[2]: *** [lib/CMakeFiles/pulsarStatic.dir/Log4cxxLogger.cc.o] Error 1 make[2]: *** Waiting for unfinished jobs.... ``` This is because the enum values renamed in https://github.com/apache/pulsar/pull/4664 are still used in `Log4cxxLogger.cc`. ### Modifications Fixed the enum values used in `Log4cxxLogger.cc` by adding the prefix `LEVEL_`. (cherry picked from commit af8ea71a)
-
由 冉小龙 提交于
Fixes #4589 Motivation When using Partition-topic, the logic of redeliver messages will not be triggered when the time of ackTimeout arrives. This is because the unAckedMessageTrackerPtr_->add(msg.getMessageId()) is not call in the listener handling of partitioned topic in cpp code (cherry picked from commit cc5f25bf)
-
由 lipenghui 提交于
Fixes #4734 ### Motivation "publish_time" is Pulsar SQL internal column, as Pulsar only stores timestamps, it doesn’t store the timezone information. Use timestamp as "publish_time" type is more correct way in Pulsar SQL. ### Modifications Change type of publish_time to timestamp. ### Verifying this change predicate of publish_time is pushdown Use `__publish_time__` to trim messages: ``` SELECT COUNT(*) FROM "sql-test-1" WHERE "__publish_time__" >= TIMESTAMP '2019-07-18 17:26:50.119' AND "__publish_time__" < TIMESTAMP '2019-07-18 17:26:51.119'; ``` ![image](https://user-images.githubusercontent.com/12592133/61447301-43835080-a983-11e9-814b-bc2b378f02b9.png) Without `__publish_time__` predicate: ``` SELECT COUNT(*) FROM "sql-test-1"; ``` ![image](https://user-images.githubusercontent.com/12592133/61447427-82190b00-a983-11e9-8d3f-3bf2a4798047.png) (cherry picked from commit 6f5416ee)
-
由 Yong Zhang 提交于
## Motivation Currently, In Pulsar Kerberos authentication, The server type part of pulsar node principle is hard coded as "broker" and "proxy". The expected principle for pulsar nodes would be like "broker/brokera.host.name@your.com" or "proxy/proxyb.host.name@your.com". But some times, user may want to re-use existing principle like "u-service/host.name@some.com", to test and play around, then "u-service" will not match service type of "broker" or "proxy", and the authentication will be rejected. This change is to remove the check of "broker" or "proxy" service type check. ## Modifaction Remove the check of "broker" or "proxy" service type check. (cherry picked from commit f5b20cd3)
-
由 Boyang Jerry Peng 提交于
* Fix: predicate pushdown for Pulsar SQL NPE * fix unit test (cherry picked from commit ac10b006)
- 27 8月, 2019 15 次提交
-
-
由 Matteo Merli 提交于
Upgrading to Mockito 2.28 and PowerMock 2.0. This a pre-step to be able to run CI with Java 11 / 12 (cherry picked from commit 108780c7)
-
由 tuteng 提交于
Fixes https://github.com/apache/pulsar/issues/4228 Master Issue: https://github.com/apache/pulsar/issues/4228 ### Motivation Use Pulsar schema in pulsar kafka client. ### Modifications Support schema of pulsar for pulsar kafka client ### Verifying this change Add Unit test (cherry picked from commit 99d3c11e)
-
由 Sijie Guo 提交于
*Motivation* Currently producers uses `System.currentTimeMillis()` as publish timestamp by default. However at some use cases, producers would like to a different way for generating publish timestamp. E.g. in a database use case, a producer might be use HLC (Hybrid Logic Clock) as publish timestamp; in integration tests, it might require the producer to use a deterministic way to generate publish timestamp. *Changes* This PR introduces a `clock` in building the client. This allows applications to override the system clock with its own implementation. *Verify the change* Add unit test to test customized clock in both batch and non-batch cases. (cherry picked from commit 7397b960)
-
由 tuteng 提交于
* ### Motivation When installing the pulsar-client-2.5.0 using the image openjdk:8-jdk-slim, for debian 10 systems, python3.7 will be installed when apt-get install python3 is executed. When installing the client using pip3, the following exception will be thrown ``` cat /etc/issue Debian GNU/Linux 10 \n \l root@40fdedf3f3c3:/# python3 Python 3.7.3 (default, Apr 3 2019, 05:39:12) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> root@40fdedf3f3c3:/# pip3 install pulsar_client-2.5.0-cp35-cp35m-manylinux1_x86_64.whl ERROR: pulsar_client-2.5.0-cp35-cp35m-manylinux1_x86_64.whl is not a supported wheel on this platform. ``` ### Modifications Install Use pip3.5 instead of pip3. * Deleute no used tools * Fix Docker command (cherry picked from commit 0a59b4ce)
-
由 Sijie Guo 提交于
*Motivation* pulsar-all and pulsar-standalone dont tag correct as pulsar image. so it fails `docker/publish.sh` *Modifications* Tag no-repo for both latest and the release version for both pulsar-all and pulsar-standalone *Verify this change* After patching this change, `docker/publish.sh` works as expected. (cherry picked from commit d2164ebf)
-
由 Ivan Kelly 提交于
feca5bbb changed topic delete logic to delete the schema when the topic is deleted (though this only seems to be enabled for idle topic GC). This exposed a bug in compatibility checking whereby if the a subscription tries to attach to the topic, even if using the same schema as had been used previously, a compatibility exception will be thrown. This is because the topic still appears to have a schema, even though there is no actual schema data, just a tombstone. I've changed the logic to return no schema if the schema read back is a tombstone. The issue doesn't affect producers because the check was already correct there. I've also updated the check for transitive compatibility to remove the prefix of schemas before the deleted schema. Previously this was throwing an NPE on the broker as it couldn't decode the deleted schema. This issue was discovered by failures in the healthcheck. The check period (5 minutes) was longer than the GC period (60 seconds). I would expect it to hit quite often in other scenarios also. (cherry picked from commit d77980d2)
-
由 Matteo Merli 提交于
* Fixed the default port for https and http in admin client * Fixed test expectation * Removed space added by mistake (cherry picked from commit c15c867c)
-
由 Matteo Merli 提交于
### Motivation Fixes #4655 Some compiler will have defined a macro for `DEBUG` and it will clash with the enum value name. Adding prefix to avoid the macro replacement. (cherry picked from commit da4e9323)
-
由 Boyang Jerry Peng 提交于
* fix issue when submitting NAR via file url * fix unit tests * add more specific errors * fix test (cherry picked from commit e8025d50)
-
由 Easyfan Zheng 提交于
C++ client producer sendAsync() method will be blocked forever, if enough batched messages sent timeout. (#4569) (#4657) (cherry picked from commit 2f686157)
-
由 massakam 提交于
### Motivation Currently, the partitioned-stats API response includes stats for each partition. However, if the number of partitions and clients is large, the size of the response will be very large. In such cases, it is useful to have a query parameter to get a response that does not include stats for each partition. ```sh $ curl -s http://localhost:8080/admin/persistent/sample/standalone/ns1/pt1/partitioned-stats | jq . { "msgRateIn": 0, "msgThroughputIn": 0, "msgRateOut": 0, "msgThroughputOut": 0, "averageMsgSize": 0, "storageSize": 0, "publishers": [], "subscriptions": { "sub1": { "msgRateOut": 0, "msgThroughputOut": 0, "msgRateRedeliver": 0, "msgBacklog": 0, "blockedSubscriptionOnUnackedMsgs": false, "msgDelayed": 0, "unackedMessages": 0, "msgRateExpired": 0, "consumers": [], "isReplicated": false } }, "replication": {}, "metadata": { "partitions": 2 }, "partitions": { "persistent://sample/standalone/ns1/pt1-partition-1": { "msgRateIn": 0, "msgThroughputIn": 0, "msgRateOut": 0, "msgThroughputOut": 0, "averageMsgSize": 0, "storageSize": 0, "publishers": [], "subscriptions": { "sub1": { "msgRateOut": 0, "msgThroughputOut": 0, "msgRateRedeliver": 0, "msgBacklog": 0, "blockedSubscriptionOnUnackedMsgs": false, "msgDelayed": 0, "unackedMessages": 0, "msgRateExpired": 0, "consumers": [], "isReplicated": false } }, "replication": {}, "deduplicationStatus": "Disabled" }, "persistent://sample/standalone/ns1/pt1-partition-0": { "msgRateIn": 0, "msgThroughputIn": 0, "msgRateOut": 0, "msgThroughputOut": 0, "averageMsgSize": 0, "storageSize": 0, "publishers": [], "subscriptions": { "sub1": { "msgRateOut": 0, "msgThroughputOut": 0, "msgRateRedeliver": 0, "msgBacklog": 0, "blockedSubscriptionOnUnackedMsgs": false, "msgDelayed": 0, "unackedMessages": 0, "msgRateExpired": 0, "consumers": [], "isReplicated": false } }, "replication": {}, "deduplicationStatus": "Disabled" } } } ``` ### Modifications Added query parameter named `perPartition` to the partitioned-stats API. The default value is true. (cherry picked from commit 977b1a85)
-
由 Rajan Dhabalia 提交于
### Motivation Broker throws NPE when pulsar-admin tries to fetch stats-internal for topic with reader. ``` Caused by: java.lang.NullPointerException at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.getProperties(ManagedCursorImpl.java:234) ~[classes/:?] at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$getInternalStats$48(PersistentTopic.java:1461) ~[classes/:?] at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_92] at org.apache.pulsar.broker.service.persistent.PersistentTopic.getInternalStats(PersistentTopic.java:1446) ~[classes/:?] at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalGetInternalStats(PersistentTopicsBase.java:621) ~[classes/:?] at org.apache.pulsar.broker.admin.v2.PersistentTopics.getInternalStats(PersistentTopics.java:430) ~[classes/:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_92] ``` (cherry picked from commit b306552f)
-