Skip to content

F
flink

doujutun3207 / flink
与 Fork 源项目一致

从无法访问的项目Fork

通知 24
Star 0
Fork 0
F
flink

体验新版 GitCode，发现更多精彩内容 >>

提交 c6209437 编写于 作者: R Robert Metzger
浏览文件
操作

Merge branch 'andrehacker-hadoop-maven-profiles'

上级 7c24622e 9964e808
隐藏空白更改
内联 并排
Showing with 290 addition and 134 deletion
.travis.yml
浏览文件 @ c6209437
......@@ -3,4 +3,10 @@ jdk:
- oraclejdk7
- openjdk7
- openjdk6
script: "mvn verify"
\ No newline at end of file
env:
- PROFILE=hadoop_v1
- PROFILE=hadoop_yarn
install: true
script: "mvn -P$PROFILE clean verify"
CONTRIBUTORS
浏览文件 @ c6209437
......@@ -13,4 +13,5 @@ Robert Metzger <metrobert@gmail.com>
Christian Richter <christian_richter@gmx.de>
Sebastian Schelter
Kostas Tzoumas
Ufuk Celebi <ufuk.celebi@campus.tu-berlin.de>
Christop Brücke
nephele/nephele-hdfs/pom.xml
浏览文件 @ c6209437
......@@ -23,24 +23,37 @@
<version>${project.version}</version>
</dependency>
<!-- Ye Olde Hadoop
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
</dependency>
-->
<!-- Coudera Hadoop (no need for hadoop-core anymore) -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
</dependency>
</dependencies>
<!-- See main pom.xml for explanation of profiles -->
<profiles>
<profile>
<id>hadoop_yarn</id>
<dependencies>
<!-- YARN -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
</dependency>
</dependencies>
</profile>
<profile>
<id>hadoop_v1</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<dependencies>
<!-- "Old" Hadoop = MapReduce v1 -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>
</project>
pact/pact-compiler/pom.xml
浏览文件 @ c6209437
......@@ -16,21 +16,18 @@
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>eu.stratosphere</groupId>
<artifactId>pact-common</artifactId>
<version>${project.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>eu.stratosphere</groupId>
<artifactId>pact-runtime</artifactId>
<version>${project.version}</version>
<scope>compile</scope>
</dependency>
</dependencies>
<reporting>
......
pact/pact-examples/pom.xml
浏览文件 @ c6209437
......@@ -21,14 +21,6 @@
<groupId>eu.stratosphere</groupId>
<artifactId>pact-common</artifactId>
<version>${project.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>eu.stratosphere</groupId>
<artifactId>pact-hbase</artifactId>
<version>${project.version}</version>
<scope>compile</scope>
</dependency>
</dependencies>
......@@ -47,31 +39,6 @@
<version>2.4</version>
<executions>
<!-- HBaseExample
<execution>
<id>HBaseExample</id>
<phase>package</phase>
<goals>
<goal>jar</goal>
</goals>
<configuration>
<classifier>HBaseExample</classifier>
<archive>
<manifestEntries>
<Pact-Assembler-Class>eu.stratosphere.pact.example.hbase.HBaseReadExample</Pact-Assembler-Class>
</manifestEntries>
</archive>
<includes>
<include>**/hbase/*.class</include>
<include>**/util/ConfigUtils.class</include>
</includes>
</configuration>
</execution>
-->
<!-- Connected Components -->
<execution>
<id>ConnectedComponents</id>
......@@ -365,4 +332,85 @@
</plugins>
</build>
<!-- See main pom.xml for explanation of profiles -->
<profiles>
<profile>
<id>hadoop_yarn</id>
<dependencies>
<dependency>
<groupId>eu.stratosphere</groupId>
<artifactId>pact-hbase</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.4</version>
<executions>
<!-- HBaseExample -->
<execution>
<id>HBaseExample</id>
<phase>package</phase>
<goals>
<goal>jar</goal>
</goals>
<configuration>
<classifier>HBaseExample</classifier>
<archive>
<manifestEntries>
<Pact-Assembler-Class>eu.stratosphere.pact.example.hbase.HBaseReadExample</Pact-Assembler-Class>
</manifestEntries>
</archive>
<includes>
<include>**/hbase/*.class</include>
<include>**/util/ConfigUtils.class</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>hadoop_v1</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<dependencies>
<!-- No extra dependencies: pact-hbase is currently not compatible with Hadoop v1 -->
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<executions>
<execution>
<id>default-compile</id>
<phase>compile</phase>
<configuration>
<excludes>
<exclude>**/hbase/*.java</exclude>
</excludes>
</configuration>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
pact/pact-hbase/pom.xml
浏览文件 @ c6209437
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<artifactId>pact</artifactId>
<groupId>eu.stratosphere</groupId>
<version>0.2-ozone</version>
</parent>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<artifactId>pact</artifactId>
<groupId>eu.stratosphere</groupId>
<version>0.2-ozone</version>
</parent>
<artifactId>pact-hbase</artifactId>
<name>pact-hbase</name>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>eu.stratosphere</groupId>
<artifactId>pact-common</artifactId>
<version>${version}</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase</artifactId>
<version>0.94.2-cdh4.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.0-cdh4.2.1</version>
<version>${hadoop.version}</version>
</dependency>
</dependencies>
</dependencies>
<!--
hadoop-client is available for yarn and non-yarn, so there is no need to use profiles
See ticket https://issues.apache.org/jira/browse/HADOOP-8009 for description of hadoop-clients
-->
<reporting>
<plugins>
......
pact/pact-hbase/src/main/java/eu/stratosphere/pact/common/io/GenericTableOutputFormat.java
浏览文件 @ c6209437
......@@ -66,6 +66,15 @@ public abstract class GenericTableOutputFormat implements OutputFormat<PactRecor
@Override
public void open(int taskNumber) throws IOException {
this.hadoopConfig = getHadoopConfig(this.config);
/**
* PLASE NOTE:
* If you are a Eclipse+Maven Integration user and you have two (or more) warnings here, please
* close the pact-hbase project OR set the maven profile to hadoop_yarn
*
* pact-hbase requires hadoop_yarn, but Eclipse is not able to parse maven profiles properly. Therefore,
* it imports the pact-hbase project even if it is not included in the standard profile (hadoop_v1)
*/
final TaskAttemptID attemptId = new TaskAttemptID(this.jtID, this.jobId, TaskType.MAP, taskNumber - 1, 0);
this.context = new org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl(this.hadoopConfig, attemptId);
......
pact/pact-runtime/pom.xml
浏览文件 @ c6209437
......@@ -16,40 +16,50 @@
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>eu.stratosphere</groupId>
<artifactId>pact-common</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>eu.stratosphere</groupId>
<artifactId>nephele-server</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>eu.stratosphere</groupId>
<artifactId>pact-array-datamodel</artifactId>
<version>${project.version}</version>
</dependency>
<!-- Ye Olde Hadoop
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
</dependency>
-->
<!-- Coudera Hadoop (no need for hadoop-core anymore) -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
</dependency>
</dependencies>
<!-- See main pom.xml for explanation of profiles -->
<profiles>
<profile>
<id>hadoop_yarn</id>
<dependencies>
<!-- YARN -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
</dependency>
</dependencies>
</profile>
<profile>
<id>hadoop_v1</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<dependencies>
<!-- "Old" Hadoop = MapReduce v1 -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
</dependency>
</dependencies>
</profile>
</profiles>
<reporting>
<plugins>
</plugins>
......@@ -57,7 +67,6 @@
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
......
pact/pom.xml
浏览文件 @ c6209437
......@@ -73,6 +73,24 @@
<module>pact-examples</module>
<module>pact-tests</module>
<module>pact-array-datamodel</module>
<module>pact-hbase</module>
</modules>
<profiles>
<profile>
<id>hadoop_yarn</id>
<modules>
<module>pact-hbase</module>
</modules>
</profile>
<profile>
<id>hadoop_v1</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<modules>
<!-- No extra modules: pact-hbase is currently not compatible with Hadoop v1 -->
</modules>
</profile>
</profiles>
</project>
pom.xml
浏览文件 @ c6209437
......@@ -33,17 +33,17 @@
</pluginRepositories>
<repositories>
<repository>
<id>cloudera-releases</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<repository>
<id>cloudera-releases</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<dependencies>
......@@ -121,39 +121,66 @@
</dependencies>
<!--
Usage of profiles for various hadoop versions:
Profile hadoop-yarn is active by default.
It will be deactivated whenever another profile becomes active in this pom
How to build for yarn using a specific version
- Call: mvn -Dhadoop.version=<version> <goal>
How to build for hadoop v1 (will use hadoop-core):
- Call: mvn -Phadoop_v1 -Dhadoop.version=<hadoop-version> <goal>
-->
<profiles>
<profile>
<id>hadoop_yarn</id>
<properties>
<hadoop.version>2.0.0-cdh4.2.1</hadoop.version>
</properties>
<dependencyManagement>
<dependencies>
<!-- YARN -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
</dependency>
</dependencies>
</dependencyManagement>
</profile>
<profile>
<id>hadoop_v1</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<hadoop.version>1.2.1</hadoop.version>
</properties>
<dependencyManagement>
<dependencies>
<!-- "Old" Hadoop = MapReduce v1 -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>${hadoop.version}</version>
</dependency>
</dependencies>
</dependencyManagement>
</profile>
</profiles>
<dependencyManagement>
<!-- this section defines the module versions that are used if nothing else is specified. -->
<dependencies>
<!-- Cloudera Hadoop -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.0.0-cdh4.2.1</version>
<type>jar</type>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.0.0-cdh4.2.1</version>
<type>jar</type>
<scope>compile</scope>
</dependency>
<!-- Ye'Olde Hadoop
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>0.20.203.0</version>
<type>jar</type>
<scope>compile</scope>
</dependency>
-->
<dependencies>
<!-- Managed dependency required for HBase in pact-hbase -->
<dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.4.3</version>
......@@ -163,7 +190,6 @@
<artifactId>slf4j-log4j12</artifactId>
<version>1.4.3</version>
</dependency>
</dependencies>
</dependencyManagement>
......@@ -389,9 +415,6 @@
</plugins>
</build>
<profiles>
</profiles>
<modules>
<module>nephele</module>
<module>pact</module>
......
stratosphere-dist/pom.xml
浏览文件 @ c6209437
......@@ -91,13 +91,32 @@
<artifactId>pact-runtime</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>eu.stratosphere</groupId>
<artifactId>pact-hbase</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
<!-- See main pom.xml for explanation of profiles -->
<profiles>
<profile>
<id>hadoop_yarn</id>
<dependencies>
<dependency>
<groupId>eu.stratosphere</groupId>
<artifactId>pact-hbase</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</profile>
<profile>
<id>hadoop_v1</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<dependencies>
<!-- No extra dependencies: pact-hbase is currently not compatible with Hadoop v1 -->
</dependencies>
</profile>
</profiles>
<build>
<plugins>
<plugin>
......
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或