Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
a9ca80bb
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
a9ca80bb
编写于
2月 20, 2019
作者:
A
Alexey Trenikhin
提交者:
Aljoscha Krettek
2月 22, 2019
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[FLINK-8354] Refactor AUTO_COMMIT config in Kafka consumers
This moves the code to a method in the base class.
上级
6051f5cb
变更
4
隐藏空白更改
内联
并排
Showing
4 changed file
with
16 addition
and
10 deletion
+16
-10
flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer010.java
...ink/streaming/connectors/kafka/FlinkKafkaConsumer010.java
+1
-4
flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.java
...link/streaming/connectors/kafka/FlinkKafkaConsumer09.java
+1
-3
flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java
...nk/streaming/connectors/kafka/FlinkKafkaConsumerBase.java
+13
-0
flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer.java
.../flink/streaming/connectors/kafka/FlinkKafkaConsumer.java
+1
-3
未找到文件。
flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer010.java
浏览文件 @
a9ca80bb
...
...
@@ -34,7 +34,6 @@ import org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema;
import
org.apache.flink.streaming.util.serialization.KeyedDeserializationSchemaWrapper
;
import
org.apache.flink.util.SerializedValue
;
import
org.apache.kafka.clients.consumer.ConsumerConfig
;
import
org.apache.kafka.clients.consumer.KafkaConsumer
;
import
org.apache.kafka.clients.consumer.OffsetAndTimestamp
;
import
org.apache.kafka.common.TopicPartition
;
...
...
@@ -190,9 +189,7 @@ public class FlinkKafkaConsumer010<T> extends FlinkKafkaConsumer09<T> {
// make sure that auto commit is disabled when our offset commit mode is ON_CHECKPOINTS;
// this overwrites whatever setting the user configured in the properties
if
(
offsetCommitMode
==
OffsetCommitMode
.
ON_CHECKPOINTS
||
offsetCommitMode
==
OffsetCommitMode
.
DISABLED
)
{
properties
.
setProperty
(
ConsumerConfig
.
ENABLE_AUTO_COMMIT_CONFIG
,
"false"
);
}
adjustAutoCommitConfig
(
properties
,
offsetCommitMode
);
return
new
Kafka010Fetcher
<>(
sourceContext
,
...
...
flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.java
浏览文件 @
a9ca80bb
...
...
@@ -243,9 +243,7 @@ public class FlinkKafkaConsumer09<T> extends FlinkKafkaConsumerBase<T> {
// make sure that auto commit is disabled when our offset commit mode is ON_CHECKPOINTS;
// this overwrites whatever setting the user configured in the properties
if
(
offsetCommitMode
==
OffsetCommitMode
.
ON_CHECKPOINTS
||
offsetCommitMode
==
OffsetCommitMode
.
DISABLED
)
{
properties
.
setProperty
(
ConsumerConfig
.
ENABLE_AUTO_COMMIT_CONFIG
,
"false"
);
}
adjustAutoCommitConfig
(
properties
,
offsetCommitMode
);
return
new
Kafka09Fetcher
<>(
sourceContext
,
...
...
flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java
浏览文件 @
a9ca80bb
...
...
@@ -54,6 +54,7 @@ import org.apache.flink.util.ExceptionUtils;
import
org.apache.flink.util.SerializedValue
;
import
org.apache.commons.collections.map.LinkedMap
;
import
org.apache.kafka.clients.consumer.ConsumerConfig
;
import
org.slf4j.Logger
;
import
org.slf4j.LoggerFactory
;
...
...
@@ -62,6 +63,7 @@ import java.util.Collection;
import
java.util.HashMap
;
import
java.util.List
;
import
java.util.Map
;
import
java.util.Properties
;
import
java.util.TreeMap
;
import
java.util.concurrent.atomic.AtomicReference
;
import
java.util.regex.Pattern
;
...
...
@@ -248,6 +250,17 @@ public abstract class FlinkKafkaConsumerBase<T> extends RichParallelSourceFuncti
this
.
useMetrics
=
useMetrics
;
}
/**
* Make sure that auto commit is disabled when our offset commit mode is ON_CHECKPOINTS.
* This overwrites whatever setting the user configured in the properties.
* @param properties - Kafka configuration properties to be adjusted
* @param offsetCommitMode offset commit mode
*/
static
void
adjustAutoCommitConfig
(
Properties
properties
,
OffsetCommitMode
offsetCommitMode
)
{
if
(
offsetCommitMode
==
OffsetCommitMode
.
ON_CHECKPOINTS
||
offsetCommitMode
==
OffsetCommitMode
.
DISABLED
)
{
properties
.
setProperty
(
ConsumerConfig
.
ENABLE_AUTO_COMMIT_CONFIG
,
"false"
);
}
}
// ------------------------------------------------------------------------
// Configuration
// ------------------------------------------------------------------------
...
...
flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer.java
浏览文件 @
a9ca80bb
...
...
@@ -221,9 +221,7 @@ public class FlinkKafkaConsumer<T> extends FlinkKafkaConsumerBase<T> {
// make sure that auto commit is disabled when our offset commit mode is ON_CHECKPOINTS;
// this overwrites whatever setting the user configured in the properties
if
(
offsetCommitMode
==
OffsetCommitMode
.
ON_CHECKPOINTS
||
offsetCommitMode
==
OffsetCommitMode
.
DISABLED
)
{
properties
.
setProperty
(
ConsumerConfig
.
ENABLE_AUTO_COMMIT_CONFIG
,
"false"
);
}
adjustAutoCommitConfig
(
properties
,
offsetCommitMode
);
return
new
KafkaFetcher
<>(
sourceContext
,
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录