Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
c6209437
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
c6209437
编写于
9月 12, 2013
作者:
R
Robert Metzger
浏览文件
操作
浏览文件
下载
差异文件
Merge branch 'andrehacker-hadoop-maven-profiles'
上级
7c24622e
9964e808
变更
11
隐藏空白更改
内联
并排
Showing
11 changed file
with
290 addition
and
134 deletion
+290
-134
.travis.yml
.travis.yml
+7
-1
CONTRIBUTORS
CONTRIBUTORS
+2
-1
nephele/nephele-hdfs/pom.xml
nephele/nephele-hdfs/pom.xml
+31
-18
pact/pact-compiler/pom.xml
pact/pact-compiler/pom.xml
+0
-3
pact/pact-examples/pom.xml
pact/pact-examples/pom.xml
+81
-33
pact/pact-hbase/pom.xml
pact/pact-hbase/pom.xml
+22
-9
pact/pact-hbase/src/main/java/eu/stratosphere/pact/common/io/GenericTableOutputFormat.java
...stratosphere/pact/common/io/GenericTableOutputFormat.java
+9
-0
pact/pact-runtime/pom.xml
pact/pact-runtime/pom.xml
+27
-18
pact/pom.xml
pact/pom.xml
+19
-1
pom.xml
pom.xml
+68
-45
stratosphere-dist/pom.xml
stratosphere-dist/pom.xml
+24
-5
未找到文件。
.travis.yml
浏览文件 @
c6209437
...
...
@@ -3,4 +3,10 @@ jdk:
-
oraclejdk7
-
openjdk7
-
openjdk6
script
:
"
mvn
verify"
\ No newline at end of file
env
:
-
PROFILE=hadoop_v1
-
PROFILE=hadoop_yarn
install
:
true
script
:
"
mvn
-P$PROFILE
clean
verify"
CONTRIBUTORS
浏览文件 @
c6209437
...
...
@@ -13,4 +13,5 @@ Robert Metzger <metrobert@gmail.com>
Christian Richter <christian_richter@gmx.de>
Sebastian Schelter
Kostas Tzoumas
Ufuk Celebi <ufuk.celebi@campus.tu-berlin.de>
Christop Brücke
nephele/nephele-hdfs/pom.xml
浏览文件 @
c6209437
...
...
@@ -23,24 +23,37 @@
<version>
${project.version}
</version>
</dependency>
<!-- Ye Olde Hadoop
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
</dependency>
-->
<!-- Coudera Hadoop (no need for hadoop-core anymore) -->
<dependency>
<groupId>
org.apache.hadoop
</groupId>
<artifactId>
hadoop-common
</artifactId>
</dependency>
<dependency>
<groupId>
org.apache.hadoop
</groupId>
<artifactId>
hadoop-hdfs
</artifactId>
</dependency>
</dependencies>
<!-- See main pom.xml for explanation of profiles -->
<profiles>
<profile>
<id>
hadoop_yarn
</id>
<dependencies>
<!-- YARN -->
<dependency>
<groupId>
org.apache.hadoop
</groupId>
<artifactId>
hadoop-common
</artifactId>
</dependency>
<dependency>
<groupId>
org.apache.hadoop
</groupId>
<artifactId>
hadoop-hdfs
</artifactId>
</dependency>
</dependencies>
</profile>
<profile>
<id>
hadoop_v1
</id>
<activation>
<activeByDefault>
true
</activeByDefault>
</activation>
<dependencies>
<!-- "Old" Hadoop = MapReduce v1 -->
<dependency>
<groupId>
org.apache.hadoop
</groupId>
<artifactId>
hadoop-core
</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>
</project>
pact/pact-compiler/pom.xml
浏览文件 @
c6209437
...
...
@@ -16,21 +16,18 @@
<packaging>
jar
</packaging>
<dependencies>
<dependency>
<groupId>
eu.stratosphere
</groupId>
<artifactId>
pact-common
</artifactId>
<version>
${project.version}
</version>
<scope>
compile
</scope>
</dependency>
<dependency>
<groupId>
eu.stratosphere
</groupId>
<artifactId>
pact-runtime
</artifactId>
<version>
${project.version}
</version>
<scope>
compile
</scope>
</dependency>
</dependencies>
<reporting>
...
...
pact/pact-examples/pom.xml
浏览文件 @
c6209437
...
...
@@ -21,14 +21,6 @@
<groupId>
eu.stratosphere
</groupId>
<artifactId>
pact-common
</artifactId>
<version>
${project.version}
</version>
<scope>
compile
</scope>
</dependency>
<dependency>
<groupId>
eu.stratosphere
</groupId>
<artifactId>
pact-hbase
</artifactId>
<version>
${project.version}
</version>
<scope>
compile
</scope>
</dependency>
</dependencies>
...
...
@@ -47,31 +39,6 @@
<version>
2.4
</version>
<executions>
<!-- HBaseExample
<execution>
<id>HBaseExample</id>
<phase>package</phase>
<goals>
<goal>jar</goal>
</goals>
<configuration>
<classifier>HBaseExample</classifier>
<archive>
<manifestEntries>
<Pact-Assembler-Class>eu.stratosphere.pact.example.hbase.HBaseReadExample</Pact-Assembler-Class>
</manifestEntries>
</archive>
<includes>
<include>**/hbase/*.class</include>
<include>**/util/ConfigUtils.class</include>
</includes>
</configuration>
</execution>
-->
<!-- Connected Components -->
<execution>
<id>
ConnectedComponents
</id>
...
...
@@ -365,4 +332,85 @@
</plugins>
</build>
<!-- See main pom.xml for explanation of profiles -->
<profiles>
<profile>
<id>
hadoop_yarn
</id>
<dependencies>
<dependency>
<groupId>
eu.stratosphere
</groupId>
<artifactId>
pact-hbase
</artifactId>
<version>
${project.version}
</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>
org.apache.maven.plugins
</groupId>
<artifactId>
maven-jar-plugin
</artifactId>
<version>
2.4
</version>
<executions>
<!-- HBaseExample -->
<execution>
<id>
HBaseExample
</id>
<phase>
package
</phase>
<goals>
<goal>
jar
</goal>
</goals>
<configuration>
<classifier>
HBaseExample
</classifier>
<archive>
<manifestEntries>
<Pact-Assembler-Class>
eu.stratosphere.pact.example.hbase.HBaseReadExample
</Pact-Assembler-Class>
</manifestEntries>
</archive>
<includes>
<include>
**/hbase/*.class
</include>
<include>
**/util/ConfigUtils.class
</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>
hadoop_v1
</id>
<activation>
<activeByDefault>
true
</activeByDefault>
</activation>
<dependencies>
<!-- No extra dependencies: pact-hbase is currently not compatible with Hadoop v1 -->
</dependencies>
<build>
<plugins>
<plugin>
<groupId>
org.apache.maven.plugins
</groupId>
<artifactId>
maven-compiler-plugin
</artifactId>
<executions>
<execution>
<id>
default-compile
</id>
<phase>
compile
</phase>
<configuration>
<excludes>
<exclude>
**/hbase/*.java
</exclude>
</excludes>
</configuration>
<goals>
<goal>
compile
</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
pact/pact-hbase/pom.xml
浏览文件 @
c6209437
<?xml version="1.0" encoding="UTF-8"?>
<project
xmlns=
"http://maven.apache.org/POM/4.0.0"
xmlns:xsi=
"http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation=
"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"
>
<modelVersion>
4.0.0
</modelVersion>
<parent>
<artifactId>
pact
</artifactId>
<groupId>
eu.stratosphere
</groupId>
<version>
0.2-ozone
</version>
</parent>
<project
xmlns=
"http://maven.apache.org/POM/4.0.0"
xmlns:xsi=
"http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation=
"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"
>
<modelVersion>
4.0.0
</modelVersion>
<parent>
<artifactId>
pact
</artifactId>
<groupId>
eu.stratosphere
</groupId>
<version>
0.2-ozone
</version>
</parent>
<artifactId>
pact-hbase
</artifactId>
<name>
pact-hbase
</name>
<packaging>
jar
</packaging>
<dependencies>
<dependency>
<groupId>
eu.stratosphere
</groupId>
<artifactId>
pact-common
</artifactId>
<version>
${version}
</version>
</dependency>
<dependency>
<groupId>
org.apache.hbase
</groupId>
<artifactId>
hbase
</artifactId>
<version>
0.94.2-cdh4.2.1
</version>
</dependency>
<dependency>
<groupId>
org.apache.hadoop
</groupId>
<artifactId>
hadoop-client
</artifactId>
<version>
2.0.0-cdh4.2.1
</version>
<version>
${hadoop.version}
</version>
</dependency>
</dependencies>
</dependencies>
<!--
hadoop-client is available for yarn and non-yarn, so there is no need to use profiles
See ticket https://issues.apache.org/jira/browse/HADOOP-8009 for description of hadoop-clients
-->
<reporting>
<plugins>
...
...
pact/pact-hbase/src/main/java/eu/stratosphere/pact/common/io/GenericTableOutputFormat.java
浏览文件 @
c6209437
...
...
@@ -66,6 +66,15 @@ public abstract class GenericTableOutputFormat implements OutputFormat<PactRecor
@Override
public
void
open
(
int
taskNumber
)
throws
IOException
{
this
.
hadoopConfig
=
getHadoopConfig
(
this
.
config
);
/**
* PLASE NOTE:
* If you are a Eclipse+Maven Integration user and you have two (or more) warnings here, please
* close the pact-hbase project OR set the maven profile to hadoop_yarn
*
* pact-hbase requires hadoop_yarn, but Eclipse is not able to parse maven profiles properly. Therefore,
* it imports the pact-hbase project even if it is not included in the standard profile (hadoop_v1)
*/
final
TaskAttemptID
attemptId
=
new
TaskAttemptID
(
this
.
jtID
,
this
.
jobId
,
TaskType
.
MAP
,
taskNumber
-
1
,
0
);
this
.
context
=
new
org
.
apache
.
hadoop
.
mapreduce
.
task
.
TaskAttemptContextImpl
(
this
.
hadoopConfig
,
attemptId
);
...
...
pact/pact-runtime/pom.xml
浏览文件 @
c6209437
...
...
@@ -16,40 +16,50 @@
<packaging>
jar
</packaging>
<dependencies>
<dependency>
<groupId>
eu.stratosphere
</groupId>
<artifactId>
pact-common
</artifactId>
<version>
${project.version}
</version>
</dependency>
<dependency>
<groupId>
eu.stratosphere
</groupId>
<artifactId>
nephele-server
</artifactId>
<version>
${project.version}
</version>
</dependency>
<dependency>
<groupId>
eu.stratosphere
</groupId>
<artifactId>
pact-array-datamodel
</artifactId>
<version>
${project.version}
</version>
</dependency>
<!-- Ye Olde Hadoop
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
</dependency>
-->
<!-- Coudera Hadoop (no need for hadoop-core anymore) -->
<dependency>
<groupId>
org.apache.hadoop
</groupId>
<artifactId>
hadoop-common
</artifactId>
</dependency>
</dependencies>
<!-- See main pom.xml for explanation of profiles -->
<profiles>
<profile>
<id>
hadoop_yarn
</id>
<dependencies>
<!-- YARN -->
<dependency>
<groupId>
org.apache.hadoop
</groupId>
<artifactId>
hadoop-common
</artifactId>
</dependency>
</dependencies>
</profile>
<profile>
<id>
hadoop_v1
</id>
<activation>
<activeByDefault>
true
</activeByDefault>
</activation>
<dependencies>
<!-- "Old" Hadoop = MapReduce v1 -->
<dependency>
<groupId>
org.apache.hadoop
</groupId>
<artifactId>
hadoop-core
</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>
<reporting>
<plugins>
</plugins>
...
...
@@ -57,7 +67,6 @@
<build>
<plugins>
<plugin>
<groupId>
org.apache.maven.plugins
</groupId>
<artifactId>
maven-surefire-plugin
</artifactId>
...
...
pact/pom.xml
浏览文件 @
c6209437
...
...
@@ -73,6 +73,24 @@
<module>
pact-examples
</module>
<module>
pact-tests
</module>
<module>
pact-array-datamodel
</module>
<module>
pact-hbase
</module>
</modules>
<profiles>
<profile>
<id>
hadoop_yarn
</id>
<modules>
<module>
pact-hbase
</module>
</modules>
</profile>
<profile>
<id>
hadoop_v1
</id>
<activation>
<activeByDefault>
true
</activeByDefault>
</activation>
<modules>
<!-- No extra modules: pact-hbase is currently not compatible with Hadoop v1 -->
</modules>
</profile>
</profiles>
</project>
pom.xml
浏览文件 @
c6209437
...
...
@@ -33,17 +33,17 @@
</pluginRepositories>
<repositories>
<repository>
<id>
cloudera-releases
</id>
<url>
https://repository.cloudera.com/artifactory/cloudera-repos
</url>
<releases>
<enabled>
true
</enabled>
</releases>
<snapshots>
<enabled>
false
</enabled>
</snapshots>
</repository>
</repositories>
<repository>
<id>
cloudera-releases
</id>
<url>
https://repository.cloudera.com/artifactory/cloudera-repos
</url>
<releases>
<enabled>
true
</enabled>
</releases>
<snapshots>
<enabled>
false
</enabled>
</snapshots>
</repository>
</repositories>
<dependencies>
...
...
@@ -121,39 +121,66 @@
</dependencies>
<!--
Usage of profiles for various hadoop versions:
Profile hadoop-yarn is active by default.
It will be deactivated whenever another profile becomes active in this pom
How to build for yarn using a specific version
- Call: mvn -Dhadoop.version=<version> <goal>
How to build for hadoop v1 (will use hadoop-core):
- Call: mvn -Phadoop_v1 -Dhadoop.version=<hadoop-version> <goal>
-->
<profiles>
<profile>
<id>
hadoop_yarn
</id>
<properties>
<hadoop.version>
2.0.0-cdh4.2.1
</hadoop.version>
</properties>
<dependencyManagement>
<dependencies>
<!-- YARN -->
<dependency>
<groupId>
org.apache.hadoop
</groupId>
<artifactId>
hadoop-common
</artifactId>
<version>
${hadoop.version}
</version>
</dependency>
<dependency>
<groupId>
org.apache.hadoop
</groupId>
<artifactId>
hadoop-hdfs
</artifactId>
<version>
${hadoop.version}
</version>
</dependency>
</dependencies>
</dependencyManagement>
</profile>
<profile>
<id>
hadoop_v1
</id>
<activation>
<activeByDefault>
true
</activeByDefault>
</activation>
<properties>
<hadoop.version>
1.2.1
</hadoop.version>
</properties>
<dependencyManagement>
<dependencies>
<!-- "Old" Hadoop = MapReduce v1 -->
<dependency>
<groupId>
org.apache.hadoop
</groupId>
<artifactId>
hadoop-core
</artifactId>
<version>
${hadoop.version}
</version>
</dependency>
</dependencies>
</dependencyManagement>
</profile>
</profiles>
<dependencyManagement>
<!-- this section defines the module versions that are used if nothing else is specified. -->
<dependencies>
<!-- Cloudera Hadoop -->
<dependency>
<groupId>
org.apache.hadoop
</groupId>
<artifactId>
hadoop-common
</artifactId>
<version>
2.0.0-cdh4.2.1
</version>
<type>
jar
</type>
<scope>
compile
</scope>
</dependency>
<dependency>
<groupId>
org.apache.hadoop
</groupId>
<artifactId>
hadoop-hdfs
</artifactId>
<version>
2.0.0-cdh4.2.1
</version>
<type>
jar
</type>
<scope>
compile
</scope>
</dependency>
<!-- Ye'Olde Hadoop
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>0.20.203.0</version>
<type>jar</type>
<scope>compile</scope>
</dependency>
-->
<dependencies>
<!-- Managed dependency required for HBase in pact-hbase -->
<dependency>
<dependency>
<groupId>
org.slf4j
</groupId>
<artifactId>
slf4j-api
</artifactId>
<version>
1.4.3
</version>
...
...
@@ -163,7 +190,6 @@
<artifactId>
slf4j-log4j12
</artifactId>
<version>
1.4.3
</version>
</dependency>
</dependencies>
</dependencyManagement>
...
...
@@ -389,9 +415,6 @@
</plugins>
</build>
<profiles>
</profiles>
<modules>
<module>
nephele
</module>
<module>
pact
</module>
...
...
stratosphere-dist/pom.xml
浏览文件 @
c6209437
...
...
@@ -91,13 +91,32 @@
<artifactId>
pact-runtime
</artifactId>
<version>
${project.version}
</version>
</dependency>
<dependency>
<groupId>
eu.stratosphere
</groupId>
<artifactId>
pact-hbase
</artifactId>
<version>
${project.version}
</version>
</dependency>
</dependencies>
<!-- See main pom.xml for explanation of profiles -->
<profiles>
<profile>
<id>
hadoop_yarn
</id>
<dependencies>
<dependency>
<groupId>
eu.stratosphere
</groupId>
<artifactId>
pact-hbase
</artifactId>
<version>
${project.version}
</version>
</dependency>
</dependencies>
</profile>
<profile>
<id>
hadoop_v1
</id>
<activation>
<activeByDefault>
true
</activeByDefault>
</activation>
<dependencies>
<!-- No extra dependencies: pact-hbase is currently not compatible with Hadoop v1 -->
</dependencies>
</profile>
</profiles>
<build>
<plugins>
<plugin>
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录