Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
0a5aebb0
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
0a5aebb0
编写于
7月 16, 2018
作者:
Z
zentol
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[FLINK-9842][rest] Pass actual configuration to BlobClient
This closes #6340.
上级
07777536
变更
3
隐藏空白更改
内联
并排
Showing
3 changed file
with
46 addition
and
18 deletion
+46
-18
flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/DispatcherRestEndpoint.java
...ache/flink/runtime/dispatcher/DispatcherRestEndpoint.java
+2
-1
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/job/JobSubmitHandler.java
...ache/flink/runtime/rest/handler/job/JobSubmitHandler.java
+8
-4
flink-runtime/src/test/java/org/apache/flink/runtime/rest/handler/job/JobSubmitHandlerTest.java
.../flink/runtime/rest/handler/job/JobSubmitHandlerTest.java
+36
-13
未找到文件。
flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/DispatcherRestEndpoint.java
浏览文件 @
0a5aebb0
...
...
@@ -92,7 +92,8 @@ public class DispatcherRestEndpoint extends WebMonitorEndpoint<DispatcherGateway
leaderRetriever
,
timeout
,
responseHeaders
,
executor
);
executor
,
clusterConfiguration
);
if
(
clusterConfiguration
.
getBoolean
(
WebOptions
.
SUBMIT_ENABLE
))
{
try
{
...
...
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/job/JobSubmitHandler.java
浏览文件 @
0a5aebb0
...
...
@@ -62,15 +62,18 @@ public final class JobSubmitHandler extends AbstractRestHandler<DispatcherGatewa
private
static
final
String
FILE_TYPE_ARTIFACT
=
"Artifact"
;
private
final
Executor
executor
;
private
final
Configuration
configuration
;
public
JobSubmitHandler
(
CompletableFuture
<
String
>
localRestAddress
,
GatewayRetriever
<?
extends
DispatcherGateway
>
leaderRetriever
,
Time
timeout
,
Map
<
String
,
String
>
headers
,
Executor
executor
)
{
Executor
executor
,
Configuration
configuration
)
{
super
(
localRestAddress
,
leaderRetriever
,
timeout
,
headers
,
JobSubmitHeaders
.
getInstance
());
this
.
executor
=
executor
;
this
.
configuration
=
configuration
;
}
@Override
...
...
@@ -99,7 +102,7 @@ public final class JobSubmitHandler extends AbstractRestHandler<DispatcherGatewa
Collection
<
Tuple2
<
String
,
Path
>>
artifacts
=
getArtifactFilesToUpload
(
requestBody
.
artifactFileNames
,
nameToFile
);
CompletableFuture
<
JobGraph
>
finalizedJobGraphFuture
=
uploadJobGraphFiles
(
gateway
,
jobGraphFuture
,
jarFiles
,
artifacts
);
CompletableFuture
<
JobGraph
>
finalizedJobGraphFuture
=
uploadJobGraphFiles
(
gateway
,
jobGraphFuture
,
jarFiles
,
artifacts
,
configuration
);
CompletableFuture
<
Acknowledge
>
jobSubmissionFuture
=
finalizedJobGraphFuture
.
thenCompose
(
jobGraph
->
gateway
.
submitJob
(
jobGraph
,
timeout
));
...
...
@@ -151,13 +154,14 @@ public final class JobSubmitHandler extends AbstractRestHandler<DispatcherGatewa
DispatcherGateway
gateway
,
CompletableFuture
<
JobGraph
>
jobGraphFuture
,
Collection
<
Path
>
jarFiles
,
Collection
<
Tuple2
<
String
,
Path
>>
artifacts
)
{
Collection
<
Tuple2
<
String
,
Path
>>
artifacts
,
Configuration
configuration
)
{
CompletableFuture
<
Integer
>
blobServerPortFuture
=
gateway
.
getBlobServerPort
(
timeout
);
return
jobGraphFuture
.
thenCombine
(
blobServerPortFuture
,
(
JobGraph
jobGraph
,
Integer
blobServerPort
)
->
{
final
InetSocketAddress
address
=
new
InetSocketAddress
(
gateway
.
getHostname
(),
blobServerPort
);
try
{
ClientUtils
.
uploadJobGraphFiles
(
jobGraph
,
jarFiles
,
artifacts
,
()
->
new
BlobClient
(
address
,
new
Configuration
()
));
ClientUtils
.
uploadJobGraphFiles
(
jobGraph
,
jarFiles
,
artifacts
,
()
->
new
BlobClient
(
address
,
configuration
));
}
catch
(
FlinkException
e
)
{
throw
new
CompletionException
(
new
RestHandlerException
(
"Could not upload job files."
,
...
...
flink-runtime/src/test/java/org/apache/flink/runtime/rest/handler/job/JobSubmitHandlerTest.java
浏览文件 @
0a5aebb0
...
...
@@ -27,6 +27,7 @@ import org.apache.flink.runtime.concurrent.FutureUtils;
import
org.apache.flink.runtime.dispatcher.DispatcherGateway
;
import
org.apache.flink.runtime.jobgraph.JobGraph
;
import
org.apache.flink.runtime.messages.Acknowledge
;
import
org.apache.flink.runtime.net.SSLUtilsTest
;
import
org.apache.flink.runtime.rest.handler.HandlerRequest
;
import
org.apache.flink.runtime.rest.handler.RestHandlerException
;
import
org.apache.flink.runtime.rest.messages.EmptyMessageParameters
;
...
...
@@ -39,12 +40,14 @@ import org.apache.flink.util.TestLogger;
import
org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpResponseStatus
;
import
org.junit.After
Class
;
import
org.junit.After
;
import
org.junit.Assert
;
import
org.junit.Before
Class
;
import
org.junit.Before
;
import
org.junit.ClassRule
;
import
org.junit.Test
;
import
org.junit.rules.TemporaryFolder
;
import
org.junit.runner.RunWith
;
import
org.junit.runners.Parameterized
;
import
java.io.IOException
;
import
java.io.ObjectOutputStream
;
...
...
@@ -57,15 +60,30 @@ import java.util.concurrent.CompletableFuture;
/**
* Tests for the {@link JobSubmitHandler}.
*/
@RunWith
(
Parameterized
.
class
)
public
class
JobSubmitHandlerTest
extends
TestLogger
{
@Parameterized
.
Parameters
(
name
=
"SSL enabled: {0}"
)
public
static
Iterable
<
Boolean
>
data
()
{
return
Arrays
.
asList
(
true
,
false
);
}
@ClassRule
public
static
final
TemporaryFolder
TEMPORARY_FOLDER
=
new
TemporaryFolder
();
private
static
BlobServer
blobServer
;
@BeforeClass
public
static
void
setup
()
throws
IOException
{
Configuration
config
=
new
Configuration
();
private
final
Configuration
configuration
;
private
BlobServer
blobServer
;
public
JobSubmitHandlerTest
(
boolean
withSsl
)
{
this
.
configuration
=
withSsl
?
SSLUtilsTest
.
createInternalSslConfigWithKeyAndTrustStores
()
:
new
Configuration
();
}
@Before
public
void
setup
()
throws
IOException
{
Configuration
config
=
new
Configuration
(
configuration
);
config
.
setString
(
BlobServerOptions
.
STORAGE_DIRECTORY
,
TEMPORARY_FOLDER
.
newFolder
().
getAbsolutePath
());
...
...
@@ -73,8 +91,8 @@ public class JobSubmitHandlerTest extends TestLogger {
blobServer
.
start
();
}
@After
Class
public
static
void
teardown
()
throws
IOException
{
@After
public
void
teardown
()
throws
IOException
{
if
(
blobServer
!=
null
)
{
blobServer
.
close
();
}
...
...
@@ -92,7 +110,8 @@ public class JobSubmitHandlerTest extends TestLogger {
()
->
CompletableFuture
.
completedFuture
(
mockGateway
),
RpcUtils
.
INF_TIMEOUT
,
Collections
.
emptyMap
(),
TestingUtils
.
defaultExecutor
());
TestingUtils
.
defaultExecutor
(),
configuration
);
JobSubmitRequestBody
request
=
new
JobSubmitRequestBody
(
jobGraphFile
.
toString
(),
Collections
.
emptyList
(),
Collections
.
emptyList
());
...
...
@@ -123,7 +142,8 @@ public class JobSubmitHandlerTest extends TestLogger {
()
->
CompletableFuture
.
completedFuture
(
mockGateway
),
RpcUtils
.
INF_TIMEOUT
,
Collections
.
emptyMap
(),
TestingUtils
.
defaultExecutor
());
TestingUtils
.
defaultExecutor
(),
configuration
);
JobSubmitRequestBody
request
=
new
JobSubmitRequestBody
(
jobGraphFile
.
getFileName
().
toString
(),
Collections
.
emptyList
(),
Collections
.
emptyList
());
...
...
@@ -151,7 +171,8 @@ public class JobSubmitHandlerTest extends TestLogger {
()
->
CompletableFuture
.
completedFuture
(
mockGateway
),
RpcUtils
.
INF_TIMEOUT
,
Collections
.
emptyMap
(),
TestingUtils
.
defaultExecutor
());
TestingUtils
.
defaultExecutor
(),
configuration
);
JobSubmitRequestBody
request
=
new
JobSubmitRequestBody
(
jobGraphFile
.
getFileName
().
toString
(),
Collections
.
emptyList
(),
Collections
.
emptyList
());
...
...
@@ -181,7 +202,8 @@ public class JobSubmitHandlerTest extends TestLogger {
()
->
CompletableFuture
.
completedFuture
(
dispatcherGateway
),
RpcUtils
.
INF_TIMEOUT
,
Collections
.
emptyMap
(),
TestingUtils
.
defaultExecutor
());
TestingUtils
.
defaultExecutor
(),
configuration
);
final
Path
jobGraphFile
=
TEMPORARY_FOLDER
.
newFile
().
toPath
();
final
Path
jarFile
=
TEMPORARY_FOLDER
.
newFile
().
toPath
();
...
...
@@ -226,7 +248,8 @@ public class JobSubmitHandlerTest extends TestLogger {
()
->
CompletableFuture
.
completedFuture
(
mockGateway
),
RpcUtils
.
INF_TIMEOUT
,
Collections
.
emptyMap
(),
TestingUtils
.
defaultExecutor
());
TestingUtils
.
defaultExecutor
(),
configuration
);
final
Path
jobGraphFile
=
TEMPORARY_FOLDER
.
newFile
().
toPath
();
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录