Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
14799738
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
14799738
编写于
5月 19, 2015
作者:
G
Gyula Fora
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[streaming] State backend configurable from flink-conf.yaml
Closes #676
上级
2d3e69a2
变更
4
隐藏空白更改
内联
并排
Showing
4 changed file
with
82 addition
and
3 deletion
+82
-3
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java
.../java/org/apache/flink/configuration/ConfigConstants.java
+15
-0
flink-dist/src/main/resources/flink-conf.yaml
flink-dist/src/main/resources/flink-conf.yaml
+15
-0
flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/graph/StreamGraph.java
...ava/org/apache/flink/streaming/api/graph/StreamGraph.java
+1
-2
flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java
.../org/apache/flink/streaming/runtime/tasks/StreamTask.java
+51
-1
未找到文件。
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java
浏览文件 @
14799738
...
...
@@ -402,6 +402,18 @@ public final class ConfigConstants {
*/
public
static
final
String
AKKA_LOOKUP_TIMEOUT
=
"akka.lookup.timeout"
;
// ----------------------------- Streaming --------------------------------
/**
* State backend for checkpoints;
*/
public
static
final
String
STATE_BACKEND
=
"state.backend"
;
/**
* Directory for saving streaming checkpoints
*/
public
static
final
String
STATE_BACKEND_FS_DIR
=
"state.backend.fs.checkpointdir"
;
// ----------------------------- Miscellaneous ----------------------------
/**
...
...
@@ -624,6 +636,9 @@ public final class ConfigConstants {
public
static
String
DEFAULT_AKKA_LOOKUP_TIMEOUT
=
"10 s"
;
// ----------------------------- Streaming Values --------------------------
public
static
String
DEFAULT_STATE_BACKEND
=
"jobmanager"
;
// ----------------------------- LocalExecution ----------------------------
...
...
flink-dist/src/main/resources/flink-conf.yaml
浏览文件 @
14799738
...
...
@@ -47,6 +47,21 @@ jobmanager.web.port: 8081
webclient.port
:
8080
#==============================================================================
# Streaming state checkpointing
#==============================================================================
# The backend that will be used to store operator state checkpoints if
# checkpointing is enabled.
#
# Supported backends: jobmanager, filesystem
state.backend
:
jobmanager
# Directory for storing checkpoints in a flink supported filesystem
#
# state.backend.fs.checkpointdir: hdfs://checkpoints
#==============================================================================
# Advanced
#==============================================================================
...
...
flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/graph/StreamGraph.java
浏览文件 @
14799738
...
...
@@ -38,7 +38,6 @@ import org.apache.flink.api.java.typeutils.MissingTypeInfo;
import
org.apache.flink.optimizer.plan.StreamingPlan
;
import
org.apache.flink.runtime.jobgraph.JobGraph
;
import
org.apache.flink.runtime.jobgraph.tasks.AbstractInvokable
;
import
org.apache.flink.runtime.state.LocalStateHandle
;
import
org.apache.flink.runtime.state.StateHandleProvider
;
import
org.apache.flink.streaming.api.collector.selector.OutputSelector
;
import
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment
;
...
...
@@ -78,7 +77,7 @@ public class StreamGraph extends StreamingPlan {
private
Map
<
Integer
,
StreamLoop
>
streamLoops
;
protected
Map
<
Integer
,
StreamLoop
>
vertexIDtoLoop
;
private
StateHandleProvider
<?>
stateHandleProvider
=
LocalStateHandle
.
createProvider
()
;
private
StateHandleProvider
<?>
stateHandleProvider
;
public
StreamGraph
(
StreamExecutionEnvironment
environment
)
{
...
...
flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java
浏览文件 @
14799738
...
...
@@ -24,12 +24,16 @@ import java.util.List;
import
org.apache.commons.collections.CollectionUtils
;
import
org.apache.commons.collections.functors.NotNullPredicate
;
import
org.apache.flink.configuration.ConfigConstants
;
import
org.apache.flink.configuration.GlobalConfiguration
;
import
org.apache.flink.runtime.event.task.TaskEvent
;
import
org.apache.flink.runtime.execution.Environment
;
import
org.apache.flink.runtime.jobgraph.tasks.AbstractInvokable
;
import
org.apache.flink.runtime.jobgraph.tasks.CheckpointCommittingOperator
;
import
org.apache.flink.runtime.jobgraph.tasks.CheckpointedOperator
;
import
org.apache.flink.runtime.jobgraph.tasks.OperatorStateCarrier
;
import
org.apache.flink.runtime.state.FileStateHandle
;
import
org.apache.flink.runtime.state.LocalStateHandle
;
import
org.apache.flink.runtime.state.StateHandle
;
import
org.apache.flink.runtime.state.StateHandleProvider
;
import
org.apache.flink.runtime.util.event.EventListener
;
...
...
@@ -76,7 +80,7 @@ public abstract class StreamTask<OUT, O extends StreamOperator<OUT>> extends Abs
this
.
userClassLoader
=
getUserCodeClassLoader
();
this
.
configuration
=
new
StreamConfig
(
getTaskConfiguration
());
this
.
context
=
createRuntimeContext
(
getEnvironment
().
getTaskName
());
this
.
stateHandleProvider
=
configuration
.
getStateHandleProvider
(
userClassLoader
);
this
.
stateHandleProvider
=
getStateHandleProvider
(
);
outputHandler
=
new
OutputHandler
<
OUT
>(
this
);
...
...
@@ -98,6 +102,52 @@ public abstract class StreamTask<OUT, O extends StreamOperator<OUT>> extends Abs
return
new
StreamingRuntimeContext
(
taskName
,
env
,
getUserCodeClassLoader
(),
getExecutionConfig
());
}
private
StateHandleProvider
<
Serializable
>
getStateHandleProvider
()
{
StateHandleProvider
<
Serializable
>
provider
=
configuration
.
getStateHandleProvider
(
userClassLoader
);
// If the user did not specify a provider in the program we try to get it from the config
if
(
provider
==
null
)
{
String
backendName
=
GlobalConfiguration
.
getString
(
ConfigConstants
.
STATE_BACKEND
,
ConfigConstants
.
DEFAULT_STATE_BACKEND
).
toUpperCase
();
StateBackend
backend
;
try
{
backend
=
StateBackend
.
valueOf
(
backendName
);
}
catch
(
Exception
e
)
{
throw
new
RuntimeException
(
backendName
+
" is not a valid state backend.\nSupported backends: jobmanager, filesystem."
);
}
switch
(
backend
)
{
case
JOBMANAGER:
LOG
.
info
(
"State backend for state checkpoints is set to jobmanager."
);
return
LocalStateHandle
.
createProvider
();
case
FILESYSTEM:
String
checkpointDir
=
GlobalConfiguration
.
getString
(
ConfigConstants
.
STATE_BACKEND_FS_DIR
,
null
);
if
(
checkpointDir
!=
null
)
{
LOG
.
info
(
"State backend for state checkpoints is set to filesystem with directory: "
+
checkpointDir
);
return
FileStateHandle
.
createProvider
(
checkpointDir
);
}
else
{
throw
new
RuntimeException
(
"For filesystem checkpointing, a checkpoint directory needs to be specified.\nFor example: \"state.backend.dir: hdfs://checkpoints\""
);
}
default
:
throw
new
RuntimeException
(
"Backend "
+
backend
+
" is not supported yet."
);
}
}
else
{
LOG
.
info
(
"Using user defined state backend for streaming checkpoitns."
);
return
provider
;
}
}
private
enum
StateBackend
{
JOBMANAGER
,
FILESYSTEM
}
protected
void
openOperator
()
throws
Exception
{
streamOperator
.
open
(
getTaskConfiguration
());
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录