Skip to content

F
flink

doujutun3207 / flink
与 Fork 源项目一致

从无法访问的项目Fork

通知 24
Star 0
Fork 0
F
flink

体验新版 GitCode，发现更多精彩内容 >>

提交 ac7d8715 编写于 作者: R Robert Metzger
浏览文件
操作

[FLINK-4895] Drop Hadoop1 support and remove related build infrastructure 

This closes #2850
上级 ae0975c1
隐藏空白更改
内联 并排
Showing with 63 addition and 637 deletion
.travis.yml
浏览文件 @ ac7d8715
......@@ -35,11 +35,6 @@ matrix:
- jdk: "oraclejdk7"
env: PROFILE="-Dhadoop.version=2.3.0 -Pflink-fast-tests-b,include-kinesis"
- jdk: "openjdk7"
env: PROFILE="-Dhadoop.profile=1 -Pflink-fast-tests-a,include-kinesis"
- jdk: "openjdk7"
env: PROFILE="-Dhadoop.profile=1 -Pflink-fast-tests-b,include-kinesis"
git:
depth: 100
......
docs/_config.yml
浏览文件 @ ac7d8715
......@@ -27,7 +27,6 @@
# we change the version for the complete docs when forking of a release branch
# etc.
version: "1.2-SNAPSHOT"
version_hadoop1: "1.2-hadoop1-SNAPSHOT"
version_short: "1.2" # Used for the top navbar w/o snapshot suffix
is_snapshot_version: true
......
docs/setup/building.md
浏览文件 @ ac7d8715
......@@ -76,31 +76,11 @@ mvn clean install
## Hadoop Versions
{% info %} Most users do not need to do this manually. The [download page]({{ site.download_url }}) contains binary packages for common Hadoop versions.
{% info %} Most users do not need to do this manually. The [download page]({{ site.download_url }}) contains binary packages for common Hadoop versions.
Flink has dependencies to HDFS and YARN which are both dependencies from [Apache Hadoop](http://hadoop.apache.org). There exist many different versions of Hadoop (from both the upstream project and the different Hadoop distributions). If you are using a wrong combination of versions, exceptions can occur.
There are two main versions of Hadoop that we need to differentiate:
- **Hadoop 1**, with all versions starting with zero or one, like *0.20*, *0.23* or *1.2.1*.
- **Hadoop 2**, with all versions starting with 2, like *2.6.0*.
The main differentiation between Hadoop 1 and Hadoop 2 is the availability of [Hadoop YARN](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html), Hadoop's cluster resource manager.
**By default, Flink is using the Hadoop 2 dependencies**.
### Hadoop 1
To build Flink for Hadoop 1, issue the following command:
~~~bash
mvn clean install -DskipTests -Dhadoop.profile=1
~~~
The `-Dhadoop.profile=1` flag instructs Maven to build Flink for Hadoop 1. Note that the features included in Flink change when using a different Hadoop profile. In particular, there is no support for YARN and HBase in Hadoop 1 builds.
### Hadoop 2.x
Hadoop 2.X versions are only supported from version 2.3.0 upwards.
Hadoop is only supported from version 2.3.0 upwards.
You can also specify a specific Hadoop version to build against:
~~~bash
......@@ -176,12 +156,3 @@ in the compiler configuration of the `pom.xml` file of the module causing the er
{% top %}
## Internals
The builds with Maven are controlled by [properties](http://maven.apache.org/pom.html#Properties) and [build profiles](http://maven.apache.org/guides/introduction/introduction-to-profiles.html). There are two profiles, one for `hadoop1` and one for `hadoop2`. When the `hadoop2` profile is enabled (default), the system will also build the YARN client.
To enable the `hadoop1` profile, set `-Dhadoop.profile=1` when building. Depending on the profile, there are two Hadoop versions, set via properties. For `hadoop1`, we use 1.2.1 by default, for `hadoop2` it is 2.3.0.
You can change these versions with the `hadoop-two.version` (or `hadoop-one.version`) property. For example `-Dhadoop-two.version=2.4.0`.
{% top %}
flink-batch-connectors/flink-hadoop-compatibility/pom.xml
浏览文件 @ ac7d8715
......@@ -62,7 +62,7 @@ under the License.
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>${shading-artifact.name}</artifactId>
<artifactId>flink-shaded-hadoop2</artifactId>
<version>${project.version}</version>
</dependency>
......
flink-batch-connectors/flink-hadoop-compatibility/src/test/java/org/apache/flink/test/hadoopcompatibility/mapreduce/example/WordCount.java
浏览文件 @ ac7d8715
......@@ -77,7 +77,6 @@ public class WordCount {
HadoopOutputFormat<Text, IntWritable> hadoopOutputFormat = new HadoopOutputFormat<Text, IntWritable>(new TextOutputFormat<Text, IntWritable>(), job);
hadoopOutputFormat.getConfiguration().set("mapreduce.output.textoutputformat.separator", " ");
hadoopOutputFormat.getConfiguration().set("mapred.textoutputformat.separator", " "); // set the value for both, since this test
// is being executed with both types (hadoop1 and hadoop2 profile)
TextOutputFormat.setOutputPath(job, new Path(outputPath));
// Output & Execute
......
flink-batch-connectors/flink-hbase/pom.xml
浏览文件 @ ac7d8715
......@@ -34,8 +34,7 @@ under the License.
<packaging>jar</packaging>
<properties>
<hbase.hadoop1.version>0.98.22-hadoop1</hbase.hadoop1.version>
<hbase.hadoop2.version>1.2.3</hbase.hadoop2.version>
<hbase.version>1.2.3</hbase.version>
</properties>
<build>
......@@ -85,7 +84,7 @@ under the License.
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>${shading-artifact.name}</artifactId>
<artifactId>flink-shaded-hadoop2</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
......@@ -198,100 +197,51 @@ under the License.
</exclusions>
</dependency>
</dependencies>
<profiles>
<profile>
<id>hadoop-1</id>
<activation>
<property>
<!-- Please do not remove the 'hadoop1' comment. See ./tools/generate_specific_pom.sh -->
<!--hadoop1--><name>hadoop.profile</name><value>1</value>
</property>
</activation>
<properties>
<hbase.version>${hbase.hadoop1.version}</hbase.version>
<!--
Required test dependencies are only available for Hadoop-2.
Disabling tests for Hadoop-1 profile.
-->
<maven.test.skip>true</maven.test.skip>
</properties>
</profile>
<profile>
<id>hadoop-2</id>
<repositories>
<repository>
<id>hadoop-2-repo2</id>
<url>https://repo.maven.apache.org/maven2</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<activation>
<property>
<!-- Please do not remove the 'hadoop2' comment. See ./tools/generate_specific_pom.sh -->
<!--hadoop2--><name>!hadoop.profile</name>
</property>
</activation>
<properties>
<hbase.version>${hbase.hadoop2.version}</hbase.version>
</properties>
<dependencies>
<!-- Test dependencies are only available for Hadoop-2. -->
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>${hbase.version}</version>
<classifier>tests</classifier>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<version>${hadoop.version}</version>
<scope>test</scope>
</dependency>
<!-- Test dependencies are only available for Hadoop-2. -->
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>${hbase.version}</version>
<classifier>tests</classifier>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-hadoop-compat</artifactId>
<version>${hbase.version}</version>
<scope>test</scope>
<type>test-jar</type>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<version>${hadoop.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-hadoop-compat</artifactId>
<version>${hbase.version}</version>
<scope>test</scope>
<type>test-jar</type>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-hadoop2-compat</artifactId>
<version>${hbase.version}</version>
<scope>test</scope>
<type>test-jar</type>
</dependency>
</dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>
</profile>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-hadoop2-compat</artifactId>
<version>${hbase.version}</version>
<scope>test</scope>
<type>test-jar</type>
</dependency>
</dependencies>
<profiles>
<profile>
<id>cdh5.1.3</id>
<properties>
<hadoop.profile>2</hadoop.profile>
<hbase.version>0.98.1-cdh5.1.3</hbase.version>
<hadoop.version>2.3.0-cdh5.1.3</hadoop.version>
<!-- Cloudera use different versions for hadoop core and commons-->
......
flink-dist/pom.xml
浏览文件 @ ac7d8715
......@@ -131,28 +131,16 @@ under the License.
<artifactId>flink-statebackend-rocksdb_2.10</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-yarn_2.10</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
<profiles>
<!-- See main pom.xml for explanation of Hadoop profiles -->
<profile>
<id>include-yarn</id>
<activation>
<property>
<!-- Please do not remove the 'hadoop2' comment. See ./tools/generate_specific_pom.sh -->
<!--hadoop2--><name>!hadoop.profile</name>
</property>
</activation>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-yarn_2.10</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</profile>
<profile>
<!-- Creates/Removes the 'build-target' symlink in the root directory (only Unix systems) -->
<id>symlink-build-target</id>
......
flink-fs-tests/pom.xml
浏览文件 @ ac7d8715
......@@ -38,7 +38,7 @@ under the License.
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>${shading-artifact.name}</artifactId>
<artifactId>flink-shaded-hadoop2</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
......
flink-java/pom.xml
浏览文件 @ ac7d8715
......@@ -43,7 +43,7 @@ under the License.
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>${shading-artifact.name}</artifactId>
<artifactId>flink-shaded-hadoop2</artifactId>
<version>${project.version}</version>
</dependency>
......
flink-mesos/pom.xml
浏览文件 @ ac7d8715
......@@ -50,7 +50,7 @@ under the License.
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>${shading-artifact.name}</artifactId>
<artifactId>flink-shaded-hadoop2</artifactId>
<version>${project.version}</version>
</dependency>
......
flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/pom.xml
浏览文件 @ ac7d8715
......@@ -166,7 +166,6 @@ under the License.
Everything else will be packaged into the fat-jar
-->
<exclude>org.apache.flink:flink-annotations</exclude>
<exclude>org.apache.flink:flink-shaded-hadoop1</exclude>
<exclude>org.apache.flink:flink-shaded-hadoop2</exclude>
<exclude>org.apache.flink:flink-shaded-curator-recipes</exclude>
<exclude>org.apache.flink:flink-core</exclude>
......
flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/pom.xml
浏览文件 @ ac7d8715
......@@ -170,7 +170,6 @@ under the License.
Everything else will be packaged into the fat-jar
-->
<exclude>org.apache.flink:flink-annotations</exclude>
<exclude>org.apache.flink:flink-shaded-hadoop1</exclude>
<exclude>org.apache.flink:flink-shaded-hadoop2</exclude>
<exclude>org.apache.flink:flink-shaded-curator-recipes</exclude>
<exclude>org.apache.flink:flink-core</exclude>
......
flink-runtime/pom.xml
浏览文件 @ ac7d8715
......@@ -52,7 +52,7 @@ under the License.
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>${shading-artifact.name}</artifactId>
<artifactId>flink-shaded-hadoop2</artifactId>
<version>${project.version}</version>
</dependency>
......
flink-runtime/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopDataOutputStream.java
浏览文件 @ ac7d8715
......@@ -19,8 +19,6 @@
package org.apache.flink.runtime.fs.hdfs;
import java.io.IOException;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import org.apache.flink.core.fs.FSDataOutputStream;
......@@ -57,78 +55,12 @@ public class HadoopDataOutputStream extends FSDataOutputStream {
@Override
public void flush() throws IOException {
if (HFLUSH_METHOD != null) {
try {
HFLUSH_METHOD.invoke(fdos);
}
catch (InvocationTargetException e) {
Throwable cause = e.getTargetException();
if (cause instanceof IOException) {
throw (IOException) cause;
}
else if (cause instanceof RuntimeException) {
throw (RuntimeException) cause;
}
else if (cause instanceof Error) {
throw (Error) cause;
}
else {
throw new IOException("Exception while invoking hflush()", cause);
}
}
catch (IllegalAccessException e) {
throw new IOException("Cannot invoke hflush()", e);
}
}
else if (HFLUSH_ERROR != null) {
if (HFLUSH_ERROR instanceof NoSuchMethodException) {
throw new UnsupportedOperationException("hflush() method is not available in this version of Hadoop.");
}
else {
throw new IOException("Cannot access hflush() method", HFLUSH_ERROR);
}
}
else {
throw new UnsupportedOperationException("hflush() is not available in this version of Hadoop.");
}
fdos.hflush();
}
@Override
public void sync() throws IOException {
if (HSYNC_METHOD != null) {
try {
HSYNC_METHOD.invoke(fdos);
}
catch (InvocationTargetException e) {
Throwable cause = e.getTargetException();
if (cause instanceof IOException) {
throw (IOException) cause;
}
else if (cause instanceof RuntimeException) {
throw (RuntimeException) cause;
}
else if (cause instanceof Error) {
throw (Error) cause;
}
else {
throw new IOException("Exception while invoking hsync()", cause);
}
}
catch (IllegalAccessException e) {
throw new IOException("Cannot invoke hsync()", e);
}
}
else if (HSYNC_ERROR != null) {
if (HSYNC_ERROR instanceof NoSuchMethodException) {
throw new UnsupportedOperationException("hsync() method is not available in this version of Hadoop.");
}
else {
throw new IOException("Cannot access hsync() method", HSYNC_ERROR);
}
}
else {
throw new UnsupportedOperationException("hsync() is not available in this version of Hadoop.");
}
fdos.hsync();
}
/**
......@@ -138,42 +70,5 @@ public class HadoopDataOutputStream extends FSDataOutputStream {
public org.apache.hadoop.fs.FSDataOutputStream getHadoopOutputStream() {
return fdos;
}
// ------------------------------------------------------------------------
// utilities to bridge hsync and hflush to hadoop, even through it is not supported in Hadoop 1
// ------------------------------------------------------------------------
private static final Method HFLUSH_METHOD;
private static final Method HSYNC_METHOD;
private static final Throwable HFLUSH_ERROR;
private static final Throwable HSYNC_ERROR;
static {
Method hflush = null;
Method hsync = null;
Throwable flushError = null;
Throwable syncError = null;
try {
hflush = org.apache.hadoop.fs.FSDataOutputStream.class.getMethod("hflush");
}
catch (Throwable t) {
flushError = t;
}
try {
hsync = org.apache.hadoop.fs.FSDataOutputStream.class.getMethod("hsync");
}
catch (Throwable t) {
syncError = t;
}
HFLUSH_METHOD = hflush;
HSYNC_METHOD = hsync;
HFLUSH_ERROR = flushError;
HSYNC_ERROR = syncError;
}
}
flink-shaded-hadoop/flink-shaded-hadoop1/pom.xml 已删除 100644 → 0
浏览文件 @ ae0975c1
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop</artifactId>
<version>1.2-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>
<artifactId>flink-shaded-hadoop1</artifactId>
<name>flink-shaded-hadoop1</name>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
<groupId>asm</groupId>
<artifactId>asm</artifactId>
</exclusion>
<exclusion>
<groupId>org.ow2.asm</groupId>
<artifactId>asm</artifactId>
</exclusion>
<exclusion>
<groupId>tomcat</groupId>
<artifactId>jasper-compiler</artifactId>
</exclusion>
<exclusion>
<groupId>tomcat</groupId>
<artifactId>jasper-runtime</artifactId>
</exclusion>
<exclusion>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty</artifactId>
</exclusion>
<exclusion>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jsp-api-2.1</artifactId>
</exclusion>
<exclusion>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jsp-2.1</artifactId>
</exclusion>
<exclusion>
<groupId>org.eclipse.jdt</groupId>
<artifactId>core</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-core</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-json</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.jettison</groupId>
<artifactId>jettison</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-server</artifactId>
</exclusion>
<exclusion>
<groupId>tomcat</groupId>
<artifactId>jasper-compiler</artifactId>
</exclusion>
<exclusion>
<groupId>tomcat</groupId>
<artifactId>jasper-runtime</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet.jsp</groupId>
<artifactId>jsp-api</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jersey.jersey-test-framework</groupId>
<artifactId>jersey-test-framework-grizzly2</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jersey.jersey-test-framework</groupId>
<artifactId>jersey-test-framework-core</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-client</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-grizzly2</artifactId>
</exclusion>
<exclusion>
<groupId>org.glassfish.grizzly</groupId>
<artifactId>grizzly-http</artifactId>
</exclusion>
<exclusion>
<groupId>org.glassfish.grizzly</groupId>
<artifactId>grizzly-framework</artifactId>
</exclusion>
<exclusion>
<groupId>org.glassfish.grizzly</groupId>
<artifactId>grizzly-http-server</artifactId>
</exclusion>
<exclusion>
<groupId>org.glassfish.grizzly</groupId>
<artifactId>grizzly-rcm</artifactId>
</exclusion>
<exclusion>
<groupId>org.glassfish.grizzly</groupId>
<artifactId>grizzly-http-servlet</artifactId>
</exclusion>
<exclusion>
<groupId>org.glassfish</groupId>
<artifactId>javax.servlet</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jersey.contribs</groupId>
<artifactId>jersey-guice</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</project>
flink-shaded-hadoop/pom.xml
浏览文件 @ ac7d8715
......@@ -35,7 +35,7 @@ under the License.
<packaging>pom</packaging>
<modules>
<module>${shading-artifact.name}</module>
<module>flink-shaded-hadoop2</module>
</modules>
<dependencies>
......@@ -50,12 +50,6 @@ under the License.
<profiles>
<profile>
<id>include-yarn-tests</id>
<activation>
<property>
<!-- Please do not remove the 'hadoop2' comment. See ./tools/generate_specific_pom.sh -->
<!--hadoop2--><name>!hadoop.profile</name>
</property>
</activation>
<modules>
<module>flink-shaded-include-yarn-tests</module>
</modules>
......
flink-streaming-connectors/flink-connector-filesystem/pom.xml
浏览文件 @ ac7d8715
......@@ -52,7 +52,7 @@ under the License.
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>${shading-artifact.name}</artifactId>
<artifactId>flink-shaded-hadoop2</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
......
flink-streaming-connectors/pom.xml
浏览文件 @ ac7d8715
......@@ -48,25 +48,11 @@ under the License.
<module>flink-connector-nifi</module>
<module>flink-connector-cassandra</module>
<module>flink-connector-redis</module>
<module>flink-connector-filesystem</module>
</modules>
<!-- See main pom.xml for explanation of profiles -->
<profiles>
<profile>
<id>hadoop-2</id>
<activation>
<property>
<!-- Please do not remove the 'hadoop2' comment. See ./tools/generate_specific_pom.sh -->
<!--hadoop2--><name>!hadoop.profile</name>
</property>
</activation>
<modules>
<!-- Include the flink-fs-tests project only for HD2.
The HDFS minicluster interfaces changed between the two versions.
-->
<module>flink-connector-filesystem</module>
</modules>
</profile>
<!--
We include the kinesis module only optionally because it contains a dependency
licenced under the "Amazon Software License".
......
flink-yarn-tests/pom.xml
浏览文件 @ ac7d8715
......@@ -81,7 +81,7 @@ under the License.
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>${shading-artifact.name}</artifactId>
<artifactId>flink-shaded-hadoop2</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
......
flink-yarn/pom.xml
浏览文件 @ ac7d8715
......@@ -52,7 +52,7 @@ under the License.
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>${shading-artifact.name}</artifactId>
<artifactId>flink-shaded-hadoop2</artifactId>
<version>${project.version}</version>
</dependency>
......
pom.xml
浏览文件 @ ac7d8715
......@@ -76,17 +76,16 @@ under the License.
<module>flink-dist</module>
<module>flink-mesos</module>
<module>flink-metrics</module>
<module>flink-yarn</module>
<module>flink-fs-tests</module>
</modules>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<!-- Mutable name of the hadoop shading artifact. The module name can contain a scala version suffix. -->
<shading-artifact.name>error</shading-artifact.name>
<!-- Internal property to reduce build times on TravisCi -->
<flink-fast-tests-pattern>never-match-me</flink-fast-tests-pattern>
<hadoop-one.version>1.2.1</hadoop-one.version>
<hadoop-two.version>2.3.0</hadoop-two.version>
<hadoop.version>2.3.0</hadoop.version>
<!-- Need to use a user property here because the surefire
forkCount is not exposed as a property. With this we can set
it on the "mvn" commandline in travis. -->
......@@ -441,43 +440,6 @@ under the License.
</properties>
</profile>
<!-- Profile to switch to Hadoop 1 -->
<profile>
<id>hadoop-1</id>
<activation>
<property>
<!-- Please do not remove the 'hadoop1' comment. See ./tools/generate_specific_pom.sh -->
<!--hadoop1--><name>hadoop.profile</name><value>1</value>
</property>
</activation>
<properties>
<hadoop.version>${hadoop-one.version}</hadoop.version>
<shading-artifact.name>flink-shaded-hadoop1</shading-artifact.name>
</properties>
</profile>
<!-- Default profile, which builds for Hadoop 2 -->
<profile>
<id>hadoop-2</id>
<activation>
<property>
<!-- Please do not remove the 'hadoop2' comment. See ./tools/generate_specific_pom.sh -->
<!--hadoop2--><name>!hadoop.profile</name>
</property>
</activation>
<properties>
<hadoop.version>${hadoop-two.version}</hadoop.version>
<shading-artifact.name>flink-shaded-hadoop2</shading-artifact.name>
</properties>
<modules>
<module>flink-yarn</module>
<!-- Include the flink-fs-tests project only for HD2.
The HDFS minicluster interfaces changed between the two versions.
-->
<module>flink-fs-tests</module>
</modules>
</profile>
<!-- Profile to deactivate the YARN tests -->
<profile>
<id>include-yarn-tests</id>
......
tools/create_release_files.sh
浏览文件 @ ac7d8715
......@@ -70,7 +70,6 @@ OLD_VERSION=${OLD_VERSION:-1.1-SNAPSHOT}
RELEASE_VERSION=${NEW_VERSION}
RELEASE_CANDIDATE=${RELEASE_CANDIDATE:-rc1}
RELEASE_BRANCH=${RELEASE_BRANCH:-master}
NEW_VERSION_HADOOP1=${NEW_VERSION_HADOOP1:-"$RELEASE_VERSION-hadoop1"}
USER_NAME=${USER_NAME:-yourapacheidhere}
MVN=${MVN:-mvn}
GPG=${GPG:-gpg}
......@@ -115,7 +114,6 @@ make_source_release() {
#change version of documentation
cd docs
perl -pi -e "s#^version: .*#version: ${NEW_VERSION}#" _config.yml
perl -pi -e "s#^version_hadoop1: .*#version_hadoop1: ${NEW_VERSION}-hadoop1#" _config.yml
perl -pi -e "s#^version_short: .*#version_short: ${NEW_VERSION}#" _config.yml
cd ..
......@@ -185,14 +183,6 @@ deploy_to_maven() {
echo "Deploying Scala 2.10 version"
cd tools && ./change-scala-version.sh 2.10 && cd ..
$MVN clean deploy -Dgpg.executable=$GPG -Prelease,docs-and-source --settings deploysettings.xml -DskipTests -Dgpg.keyname=$GPG_KEY -Dgpg.passphrase=$GPG_PASSPHRASE -DretryFailedDeploymentCount=10
echo "Deploying Scala 2.10 / hadoop 1 version"
../generate_specific_pom.sh $NEW_VERSION $NEW_VERSION_HADOOP1 pom.xml
sleep 4
$MVN clean deploy -Dgpg.executable=$GPG -Prelease,docs-and-source --settings deploysettings.xml -DskipTests -Dgpg.keyname=$GPG_KEY -Dgpg.passphrase=$GPG_PASSPHRASE -DretryFailedDeploymentCount=10
}
copy_data() {
......@@ -211,7 +201,6 @@ prepare
make_source_release
make_binary_release "hadoop1" "-Dhadoop.profile=1" 2.10
make_binary_release "hadoop2" "" 2.10
make_binary_release "hadoop24" "-Dhadoop.version=2.4.1" 2.10
make_binary_release "hadoop26" "-Dhadoop.version=2.6.3" 2.10
......
tools/deploy_to_maven.sh
浏览文件 @ ac7d8715
......@@ -75,13 +75,8 @@ rm -rf dummy-lifecycle-mapping-plugin
CURRENT_FLINK_VERSION=`getVersion`
if [[ "$CURRENT_FLINK_VERSION" == *-SNAPSHOT ]]; then
CURRENT_FLINK_VERSION_HADOOP1=${CURRENT_FLINK_VERSION/-SNAPSHOT/-hadoop1-SNAPSHOT}
else
CURRENT_FLINK_VERSION_HADOOP1="$CURRENT_FLINK_VERSION-hadoop1"
fi
echo "detected current version as: '$CURRENT_FLINK_VERSION' ; hadoop1: $CURRENT_FLINK_VERSION_HADOOP1 "
echo "detected current version as: '$CURRENT_FLINK_VERSION'"
#
# This script deploys our project to sonatype SNAPSHOTS.
......@@ -92,13 +87,6 @@ if [[ $CURRENT_FLINK_VERSION == *SNAPSHOT* ]] ; then
MVN_SNAPSHOT_OPTS="-B -Pdocs-and-source -DskipTests -Drat.skip=true -Drat.ignoreErrors=true \
-DretryFailedDeploymentCount=10 --settings deploysettings.xml clean deploy"
# Deploy hadoop v1 to maven
echo "Generating poms for hadoop1"
./tools/generate_specific_pom.sh $CURRENT_FLINK_VERSION $CURRENT_FLINK_VERSION_HADOOP1 pom.hadoop1.xml
mvn -f pom.hadoop1.xml ${MVN_SNAPSHOT_OPTS}
# deploy to s3
deploy_to_s3 $CURRENT_FLINK_VERSION "hadoop1"
# hadoop2 scala 2.10
echo "deploy standard version (hadoop2) for scala 2.10"
mvn ${MVN_SNAPSHOT_OPTS}
......
tools/generate_specific_pom.sh 已删除 100755 → 0
浏览文件 @ ae0975c1
#!/usr/bin/env bash
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
#
# Inspired by and modified from:
# https://github.com/apache/hbase/blob/trunk/dev-support/generate-hadoopX-poms.sh
#
function usage {
echo "Usage: $0 CURRENT_VERSION NEW_VERSION [POM_NAME]"
echo "For example, $0 0.8-incubating-SNAPSHOT 0.8-hadoop1-incubating-SNAPSHOT"
echo "Presumes VERSION has hadoop1 or hadoop2 in it. POM_NAME is optional and"
echo "allows to specify a different name for the generated pom."
exit 1
}
if [[ "$#" -lt 2 ]]; then usage; fi
old_version="$1"
new_version="$2"
new_pom_name="$3"
# Get hadoop version from the new Flink version
hadoop_version=`echo "$new_version" | sed -n 's/.*\(hadoop[12]\).*/\1/p'`
if [[ -z $hadoop_version ]]; then usage ; fi
echo "hadoop version $hadoop_version"
here="`dirname \"$0\"`" # relative
here="`( cd \"$here\" && pwd )`" # absolutized and normalized
if [ -z "$here" ] ; then
# error; for some reason, the path is not accessible
# to the script (e.g. permissions re-evaled after suid)
exit 1 # fail
fi
flink_home="`dirname \"$here\"`"
hadoop1=
hadoop2=
default='<name>!hadoop.profile<\/name>'
notdefault='<name>hadoop.profile<\/name>'
case "${hadoop_version}" in
hadoop1)
hadoop1="${default}"
hadoop2="${notdefault}<value>2<\/value>"
;;
hadoop2)
hadoop1="${notdefault}<value>1<\/value>"
hadoop2="${default}"
;;
*) echo "Unknown ${hadoop_version}"
usage
;;
esac
nupom=$new_pom_name
if [[ -z "$new_pom_name" ]]; then
nupom="pom.${hadoop_version}.xml"
fi
echo "Using $nupom as name for the generated pom file."
# export relevant variables for find command subshells
export old_version
export new_version
export hadoop1
export hadoop2
export nupom
# paths may contain spaces
find "$flink_home" -name pom.xml -exec bash -c '
p="$0"
# write into tmp file because in-place replacement is not possible (if nupom="pom.xml")
tmp_nuname1="`dirname "$p"`/__generate_specific_pom_tmp1"
tmp_nuname2="`dirname "$p"`/__generate_specific_pom_tmp2"
nuname="`dirname "$p"`/${nupom}"
# Now we do search and replace of explicit strings. The best way of
# seeing what the below does is by doing a diff between the original
# pom and the generated pom (pom.hadoop1.xml or pom.hadoop2.xml). We
# replace the version string in all poms, we change modules to
# include reference to the non- standard pom name, we adjust
# relative paths so child modules can find the parent pom, and we
# enable/disable hadoop 1 and hadoop 2 profiles as appropriate
# removing a comment string too. We output the new pom beside the
# original.
# To avoid accidentally replace version numbers in our dependencies
# sharing the version number with the current release use the following.
echo "p=$p, old_version=${old_version}, new_version=$new_version"
perl -0777 -pe "s:<groupId>org.apache.flink</groupId>\n([\t ]*<artifactId>([a-z]+-)+[a-z0-9\.\_]+</artifactId>\n[\t ]*)<version>${old_version}</version>:<groupId>org.apache.flink</groupId>\n\1<version>${new_version}</version>:g" "$p" > "$tmp_nuname1"
# replace the version also in the quickstart poms (so that the hadoop1 quickstart creates an hadoop1 project)
perl -0777 -pe "s:<flink.version>${old_version}</flink.version>:<flink.version>${new_version}</flink.version>:g" "$tmp_nuname1" > "$tmp_nuname2"
# Alternatively when no version collisions are present this is enough:
# sed -e "s/${old_version}/${new_version}/" $p > "$tmp_nuname1"
sed -e "s/\(<module>[^<]*\)/\1\/${nupom}/" \
-e "s/\(relativePath>\.\.\)/\1\/${nupom}/" \
-e "s/<!--hadoop1-->.*name>.*/${hadoop1}/" \
-e "s/<!--hadoop2-->.*name>.*/${hadoop2}/" \
"$tmp_nuname2" > "$tmp_nuname1"
rm "$tmp_nuname2"
mv "$tmp_nuname1" "$nuname"
' "{}" \; # pass file name as argument
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或