Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
7ddfb16a
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
7ddfb16a
编写于
12月 21, 2010
作者:
F
Fabian Hueske
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
- moved filesystem classes into designated package
上级
717a3775
变更
5
隐藏空白更改
内联
并排
Showing
5 changed file
with
469 addition
and
0 deletion
+469
-0
pact/pact-tests/src/test/java/eu/stratosphere/pact/test/util/filesystem/ExternalDFSProvider.java
...sphere/pact/test/util/filesystem/ExternalDFSProvider.java
+46
-0
pact/pact-tests/src/test/java/eu/stratosphere/pact/test/util/filesystem/FilesystemProvider.java
...osphere/pact/test/util/filesystem/FilesystemProvider.java
+33
-0
pact/pact-tests/src/test/java/eu/stratosphere/pact/test/util/filesystem/HDFSProvider.java
.../stratosphere/pact/test/util/filesystem/HDFSProvider.java
+149
-0
pact/pact-tests/src/test/java/eu/stratosphere/pact/test/util/filesystem/LocalFSProvider.java
...ratosphere/pact/test/util/filesystem/LocalFSProvider.java
+124
-0
pact/pact-tests/src/test/java/eu/stratosphere/pact/test/util/filesystem/MiniDFSProvider.java
...ratosphere/pact/test/util/filesystem/MiniDFSProvider.java
+117
-0
未找到文件。
pact/pact-tests/src/test/java/eu/stratosphere/pact/test/util/filesystem/ExternalDFSProvider.java
0 → 100644
浏览文件 @
7ddfb16a
/***********************************************************************************************************************
*
* Copyright (C) 2010 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.pact.test.util.filesystem
;
import
java.io.IOException
;
import
org.apache.hadoop.conf.Configuration
;
import
org.apache.hadoop.fs.FileSystem
;
import
org.apache.hadoop.fs.Path
;
public
class
ExternalDFSProvider
extends
HDFSProvider
{
public
ExternalDFSProvider
(
String
configDir
)
{
super
(
configDir
);
}
public
void
start
()
throws
Exception
{
Configuration
config
=
new
Configuration
(
false
);
config
.
addResource
(
new
Path
(
configDir
+
"/hadoop-default.xml"
));
config
.
addResource
(
new
Path
(
configDir
+
"/hadoop-site.xml"
));
hdfs
=
FileSystem
.
get
(
config
);
}
public
void
stop
()
{
try
{
hdfs
.
close
();
}
catch
(
IOException
e
)
{
e
.
printStackTrace
();
}
hdfs
=
null
;
}
}
pact/pact-tests/src/test/java/eu/stratosphere/pact/test/util/filesystem/FilesystemProvider.java
0 → 100644
浏览文件 @
7ddfb16a
package
eu.stratosphere.pact.test.util.filesystem
;
import
java.io.IOException
;
import
java.io.InputStream
;
import
java.io.OutputStream
;
public
interface
FilesystemProvider
{
public
void
start
()
throws
Exception
;
public
void
stop
();
public
boolean
createFile
(
String
fileName
,
String
fileContent
)
throws
IOException
;
public
boolean
copyFile
(
String
localFile
,
String
hdfsFile
)
throws
IOException
;
public
boolean
createDir
(
String
dirName
)
throws
IOException
;
public
boolean
delete
(
String
path
,
boolean
recursive
)
throws
IOException
;
public
OutputStream
getOutputStream
(
String
file
)
throws
IOException
;
public
InputStream
getInputStream
(
String
file
)
throws
IOException
;
public
String
getTempDirPath
();
public
String
[]
listFiles
(
String
dir
)
throws
IOException
;
public
boolean
isDir
(
String
file
)
throws
IOException
;
public
String
getURIPrefix
();
}
pact/pact-tests/src/test/java/eu/stratosphere/pact/test/util/filesystem/HDFSProvider.java
0 → 100644
浏览文件 @
7ddfb16a
/***********************************************************************************************************************
*
* Copyright (C) 2010 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.pact.test.util.filesystem
;
import
java.io.File
;
import
java.io.FileReader
;
import
java.io.IOException
;
import
java.io.InputStream
;
import
java.io.OutputStream
;
import
java.io.OutputStreamWriter
;
import
java.io.Writer
;
import
java.util.ArrayList
;
import
org.apache.hadoop.fs.FileStatus
;
import
org.apache.hadoop.fs.FileSystem
;
import
org.apache.hadoop.fs.Path
;
public
abstract
class
HDFSProvider
implements
FilesystemProvider
{
protected
FileSystem
hdfs
;
protected
final
String
configDir
;
abstract
public
void
start
()
throws
Exception
;
abstract
public
void
stop
();
protected
HDFSProvider
(
String
config
)
{
this
.
configDir
=
config
;
}
public
FileSystem
getFileSystem
()
{
return
hdfs
;
}
public
String
getConfigDir
()
{
return
configDir
;
}
public
OutputStream
getOutputStream
(
String
file
)
throws
IOException
{
return
hdfs
.
create
(
new
org
.
apache
.
hadoop
.
fs
.
Path
(
file
));
}
public
InputStream
getInputStream
(
String
file
)
throws
IOException
{
return
hdfs
.
open
(
new
org
.
apache
.
hadoop
.
fs
.
Path
(
file
));
}
public
String
getTempDirPath
()
{
return
"/user/"
+
hdfs
.
getHomeDirectory
().
getName
();
}
public
boolean
createDir
(
String
dirName
)
throws
IOException
{
if
(!
getFileSystem
().
mkdirs
(
new
Path
(
dirName
)))
{
return
false
;
}
while
(!
getFileSystem
().
exists
(
new
Path
(
dirName
)))
{
try
{
Thread
.
sleep
(
100
);
}
catch
(
InterruptedException
e
)
{
e
.
printStackTrace
();
}
}
return
true
;
}
public
boolean
delete
(
String
path
,
boolean
recursive
)
throws
IOException
{
if
(!
getFileSystem
().
delete
(
new
Path
(
path
),
recursive
))
{
return
false
;
}
while
(
getFileSystem
().
exists
(
new
Path
(
path
)))
{
try
{
Thread
.
sleep
(
100
);
}
catch
(
InterruptedException
e
)
{
e
.
printStackTrace
();
}
}
return
true
;
}
public
boolean
createFile
(
String
fileName
,
String
fileContent
)
throws
IOException
{
if
(
getFileSystem
().
exists
(
new
Path
(
fileName
)))
{
return
false
;
}
OutputStream
os
=
this
.
getOutputStream
(
fileName
);
Writer
wr
=
new
OutputStreamWriter
(
os
);
wr
.
write
(
fileContent
);
wr
.
close
();
while
(!
getFileSystem
().
exists
(
new
Path
(
fileName
)))
{
try
{
Thread
.
sleep
(
100
);
}
catch
(
InterruptedException
e
)
{
e
.
printStackTrace
();
}
}
return
true
;
}
public
boolean
copyFile
(
String
localFile
,
String
hdfsFile
)
throws
IOException
{
if
(
getFileSystem
().
exists
(
new
Path
(
hdfsFile
)))
{
return
false
;
}
FileReader
fr
=
new
FileReader
(
new
File
(
localFile
));
OutputStream
os
=
this
.
getOutputStream
(
hdfsFile
);
Writer
wr
=
new
OutputStreamWriter
(
os
);
while
(
fr
.
ready
())
{
wr
.
write
(
fr
.
read
());
}
wr
.
close
();
fr
.
close
();
return
true
;
}
public
String
[]
listFiles
(
String
dir
)
throws
IOException
{
FileStatus
[]
fss
=
hdfs
.
listStatus
(
new
Path
(
dir
));
ArrayList
<
String
>
fileList
=
new
ArrayList
<
String
>(
fss
.
length
);
for
(
FileStatus
fs
:
fss
)
{
fileList
.
add
(
fs
.
getPath
().
toString
());
}
return
fileList
.
toArray
(
new
String
[
1
]);
}
public
boolean
isDir
(
String
file
)
throws
IOException
{
return
hdfs
.
getFileStatus
(
new
Path
(
file
)).
isDir
();
}
public
String
getURIPrefix
()
{
return
hdfs
.
getUri
().
toString
();
}
}
pact/pact-tests/src/test/java/eu/stratosphere/pact/test/util/filesystem/LocalFSProvider.java
0 → 100644
浏览文件 @
7ddfb16a
package
eu.stratosphere.pact.test.util.filesystem
;
import
java.io.File
;
import
java.io.FileInputStream
;
import
java.io.FileOutputStream
;
import
java.io.FileReader
;
import
java.io.FileWriter
;
import
java.io.IOException
;
import
java.io.InputStream
;
import
java.io.OutputStream
;
public
class
LocalFSProvider
implements
FilesystemProvider
{
public
boolean
createDir
(
String
dirName
)
throws
IOException
{
File
f
=
new
File
(
dirName
);
if
(
f
.
exists
())
{
return
false
;
}
return
f
.
mkdir
();
}
public
boolean
createFile
(
String
fileName
,
String
fileContent
)
throws
IOException
{
File
f
=
new
File
(
fileName
);
if
(
f
.
exists
())
{
return
false
;
}
FileWriter
fw
=
new
FileWriter
(
f
);
fw
.
write
(
fileContent
);
fw
.
close
();
return
true
;
}
public
boolean
copyFile
(
String
source
,
String
target
)
throws
IOException
{
File
t
=
new
File
(
target
);
if
(
t
.
exists
())
{
return
false
;
}
File
s
=
new
File
(
source
);
FileWriter
tw
=
new
FileWriter
(
t
);
FileReader
sr
=
new
FileReader
(
s
);
char
[]
buffer
=
new
char
[
1024
];
int
copiedBytes
=
sr
.
read
(
buffer
);
while
(
copiedBytes
>
-
1
)
{
tw
.
write
(
buffer
,
0
,
copiedBytes
);
copiedBytes
=
sr
.
read
(
buffer
);
}
sr
.
close
();
tw
.
close
();
return
true
;
}
public
boolean
delete
(
String
path
,
boolean
recursive
)
throws
IOException
{
File
f
=
new
File
(
path
);
if
(
f
.
isDirectory
()
&&
recursive
)
{
for
(
String
c
:
f
.
list
())
{
this
.
delete
(
c
,
true
);
}
f
.
delete
();
return
true
;
}
else
if
(
f
.
isDirectory
()
&&
!
recursive
)
{
return
false
;
}
else
{
f
.
delete
();
return
true
;
}
}
public
InputStream
getInputStream
(
String
file
)
throws
IOException
{
return
new
FileInputStream
(
file
);
}
public
OutputStream
getOutputStream
(
String
file
)
throws
IOException
{
return
new
FileOutputStream
(
file
);
}
public
String
getTempDirPath
()
{
return
System
.
getProperty
(
"java.io.tmpdir"
);
}
@Override
public
void
start
()
throws
Exception
{
}
@Override
public
void
stop
()
{
}
@Override
public
boolean
isDir
(
String
file
)
throws
IOException
{
return
(
new
File
(
file
)).
isDirectory
();
}
@Override
public
String
[]
listFiles
(
String
dir
)
throws
IOException
{
File
f
=
new
File
(
dir
);
if
(!
f
.
isDirectory
()){
return
null
;
}
else
{
return
f
.
list
();
}
}
@Override
public
String
getURIPrefix
()
{
return
"file://"
;
}
}
pact/pact-tests/src/test/java/eu/stratosphere/pact/test/util/filesystem/MiniDFSProvider.java
0 → 100644
浏览文件 @
7ddfb16a
/***********************************************************************************************************************
*
* Copyright (C) 2010 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.pact.test.util.filesystem
;
import
java.io.File
;
import
java.io.IOException
;
import
org.apache.hadoop.conf.Configuration
;
import
org.apache.hadoop.hdfs.MiniDFSCluster
;
import
eu.stratosphere.pact.test.util.FileWriter
;
/**
* @author Erik Nijkamp
*/
public
class
MiniDFSProvider
extends
HDFSProvider
{
private
MiniDFSCluster
cluster
;
private
final
static
int
NAME_NODE_PORT
=
9000
;
public
MiniDFSProvider
()
{
super
(
getTempDir
()
+
"/config"
);
}
public
void
start
()
throws
Exception
{
deleteDir
(
getTempDir
());
// create dirs
new
FileWriter
().
dir
(
getTempDir
());
new
FileWriter
().
dir
(
getLogDir
());
new
FileWriter
().
dir
(
getDfsDir
());
new
FileWriter
().
dir
(
getConfigDir
());
// set system properies needed by the MiniDFSCluster
System
.
setProperty
(
"hadoop.log.dir"
,
getLogDir
());
System
.
setProperty
(
"test.build.data"
,
getDfsDir
());
// init hdfs cluster
cluster
=
new
MiniDFSCluster
(
NAME_NODE_PORT
,
new
Configuration
(),
1
,
true
,
true
,
null
,
null
);
hdfs
=
cluster
.
getFileSystem
();
// write hdfs config files needed by nephele in temp folder
new
FileWriter
().
dir
(
getConfigDir
()).
file
(
"hadoop-site.xml"
).
write
(
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>"
,
"<?xml-stylesheet href=\"configuration.xsl\" type=\"text/xsl\"?>"
,
"<configuration>"
,
" <property>"
,
" <name>fs.default.name</name>"
,
" <value>"
+
"hdfs://localhost:"
+
cluster
.
getNameNodePort
()
+
"</value>"
,
" </property>"
,
" <property>"
,
" <name>dfs.name.dir</name>"
,
" <value>"
+
getDfsDir
()
+
"/name</value>"
,
" </property>"
,
" <property>"
,
" <name>dfs.data.dir</name>"
,
" <value>"
+
getDfsDir
()
+
"/data</value>"
,
" </property>"
,
"</configuration>"
).
close
().
file
(
"hadoop-default.xml"
).
write
(
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>"
,
"<?xml-stylesheet href=\"configuration.xsl\" type=\"text/xsl\"?>"
,
"<configuration>"
,
"</configuration>"
)
.
close
();
}
public
void
stop
()
{
try
{
hdfs
.
close
();
}
catch
(
IOException
e1
)
{
e1
.
printStackTrace
();
}
cluster
.
shutdown
();
deleteDir
(
getTempDir
());
try
{
Thread
.
sleep
(
10000
);
}
catch
(
InterruptedException
e
)
{
}
}
private
static
String
getTempDir
()
{
return
System
.
getProperty
(
"user.dir"
)
+
"/tmp"
;
}
private
static
String
getDfsDir
()
{
return
getTempDir
()
+
"/dfs"
;
}
private
static
String
getLogDir
()
{
return
getTempDir
()
+
"/logs"
;
}
private
boolean
deleteDir
(
String
dir
)
{
return
deleteDir
(
new
File
(
dir
));
}
private
boolean
deleteDir
(
File
dir
)
{
if
(
dir
.
isDirectory
())
{
String
[]
childrens
=
dir
.
list
();
for
(
String
child
:
childrens
)
{
boolean
success
=
deleteDir
(
new
File
(
dir
,
child
));
if
(!
success
)
{
return
false
;
}
}
}
return
dir
.
delete
();
}
}
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录