2021-05-06 21:27:21

doc/flink-17-doc-zh/1.md

+# Apache Flink 文档
+
+本文档适用于Apache Flink 1.7版本。该内容构建于:2019-02-16。
+
+Apache Flink是一个用于分布式流和批处理数据的开源平台。Flink的核心是一个流数据流引擎，它为数据流上的分布式计算提供数据分发、通信和容错功能。Flink在流引擎之上构建批处理、覆盖本地迭代支持、托管内存和程序优化。
+
+## 第一步
+
+*   **概念**: 从Flink的基本概念开始 [数据流编程模型](concepts/programming-model.html) 和 [分布式运行环境](concepts/runtime.html). 这将帮助您理解文档的其他部分，包括设置和编程指南。我们建议您先阅读这些部分。
+
+*   **教程**:
+    *   [实现并运行一个DataStream应用程序](./tutorials/datastream_api.html)
+    *   [设置本地Flink集群](./tutorials/local_setup.html)
+*   **编程指南**: 你可以阅读我们的指南 [API的基本概念](dev/api_concepts.html) 和 [DataStream API](dev/datastream_api.html) 或者 [DataSet API](dev/batch/index.html) 学习如何编写您的第一个Flink程序。
+
+## 部署
+
+在你的Flink作业投入生产之前, 阅读 [生产准备检查表](ops/production_ready.html).
+
+## 发布说明
+
+发布说明涵盖了Flink版本之间的重要更改。如果您打算将Flink设置升级到一个较新的版本，请仔细阅读这些说明。
+
+*   [Flink 1.7 发布说明](release-notes/flink-1.7.html).
+*   [Flink 1.6 发布说明](release-notes/flink-1.6.html).
+*   [Flink 1.5 发布说明](release-notes/flink-1.5.html).
+
+## 外部资源
+
+*   **Flink 历程**: 过去会议的发言可在[Flink Forward](http://flink-forward.org/) 网站和 [YouTube](https://www.youtube.com/channel/UCY8_lgiZLZErZPF47a2hXMA). [Robust Stream Processing with Apache Flink](http://2016.flink-forward.org/kb_sessions/robust-stream-processing-with-apache-flink/) 这是认知Flink很好的起点.
+
+*   **培训**:  在[training materials](http://training.data-artisans.com/) 该链接提供教案、实例联系、示例解决方案；
+
+*   **博客**: 在 [Apache Flink](https://flink.apache.org/blog/) 和 [data Artisans](https://data-artisans.com/blog/) 经常发布滚鱼Flink的深入技术文章。

doc/flink-17-doc-zh/10.md

+
+
+# Scala项目模板
+
+## 构建工具
+
+Flink项目可以使用不同的构建工具来构建。为了快速启动，Flink为以下构建工具提供了项目模板:
+
+*   [SBT](#sbt)
+*   [Maven](#maven)
+
+这些模板帮助您设置项目结构并创建初始构建文件。
+
+## SBT
+
+### 创建项目
+
+您可以通过以下两种方法来构建一个新项目:
+
+*   [Use the **sbt template**](#sbt_template)
+*   [Run the **quickstart script**](#quickstart-script-sbt)
+
+
+
+```
+ $ sbt new tillrohrmann/flink-project.g8 
+```
+
+
+
+这将提示您输入几个参数(项目名称、flink版本……)，然后从 [flink-project template](https://github.com/tillrohrmann/flink-project.g8)创建一个flink项目。您需要sbt >= 0.13.13来执行这个命令。如果有必要，您可以按照[installation guide](http://www.scala-sbt.org/download.html) 获取它。
+
+
+
+```
+ $ bash <(curl https://flink.apache.org/q/sbt-quickstart.sh) 
+```
+
+
+
+这将在**specified** 项目目录中创建一个Flink项目。
+
+### 构建项目
+
+为了构建您的项目，您只需发出 `sbt clean assembly` 命令。这将创建**your-project-name-assembly-0.1-SNAPSHOT.jar**中的 **target/scala_your-major-scala-version/**。
+
+### 运行项目
+
+为了运行您的项目，您必须发出“sbt run”命令。
+
+默认情况下，这将在运行`sbt`的JVM中运行作业。为了在不同的JVM中运行作业，请在`build.sbt`中添加以下行
+
+
+
+```
+fork in run := true
+```
+
+
+
+#### IntelliJ
+
+我们建议使用[IntelliJ](https://www.jetbrains.com/idea/)进行Flink作业开发。为了开始，您必须将新创建的项目导入IntelliJ。你可以通过`File -&gt; New -&gt; Project from Existing Sources...`然后选择项目的目录。IntelliJ会自动检测`build.sbt` 文件和设置一切。
+
+为了运行Flink作业，建议选择`mainRunner` 模块作为 **Run/Debug Configuration**的类路径。这将确保所有设置为 _provided_ 的依赖项在执行时都是可用的。您可以通过**Run/Debug Configurations** `Run -&gt; Edit Configurations...`，然后从 _Use classpath of module_ 类路径中选择`mainRunner`。
+
+#### Eclipse
+
+为了将新创建的项目导入[Eclipse](https://eclipse.org/)，首先必须为其创建Eclipse项目文件。这些项目文件可以通过[sbteclipse](https://github.com/typesafehub/sbteclipse) 插件创建。在`PROJECT_DIR/project/plugins.sbt`中添加以下行的文件:
+
+
+
+```
+addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0")
+```
+
+
+
+在`sbt`中，使用以下命令创建Eclipse项目文件
+
+
+
+```
+> eclipse
+```
+
+
+
+现在您可以通过`File -&gt; Import... -&gt; Existing Projects into Workspace` 然后选择项目目录。
+
+## Maven
+
+### 需求
+
+唯一的要求是**Maven 3.0.4**(或更高)和 **Java 8.x** 安装。
+
+### 创建项目
+
+使用以下命令之一**create a project**:
+
+*   [Use **Maven archetypes**](#maven-archetype)
+*   [Run the **quickstart script**](#quickstart-script)
+
+
+
+```
+ $ mvn archetype:generate                               \
+      -DarchetypeGroupId=org.apache.flink              \
+      -DarchetypeArtifactId=flink-quickstart-scala     \
+      -DarchetypeVersion=1.7.1 
+```
+
+
+
+这允许您将新创建的项目命名为。它会交互式地询问groupId、artifactId和包名。
+
+
+
+```
+ $ curl https://flink.apache.org/q/quickstart-scala.sh | bash -s 1.7.1
+```
+
+
+
+### 检查项目
+
+您的工作目录中将有一个新目录。如果您使用了 _curl_ 方法，则该目录称为 `quickstart`。否则，它有您的`artifactId`的名称:
+
+
+
+```
+$ tree quickstart/
+quickstart/
+├── pom.xml
+└── src
+    └── main
+        ├── resources
+        │   └── log4j.properties
+        └── scala
+            └── org
+                └── myorg
+                    └── quickstart
+                        ├── BatchJob.scala
+                        └── StreamingJob.scala
+```
+
+
+
+示例项目是一个**Maven project**，其中包含两个类:_StreamingJob_ 和 _BatchJob_ 是_ DataStream_ 和 _DataSet_ 程序的基本框架程序。_main_ 方法是程序的入口点，用于ide内测试/执行和适当的部署。
+
+我们建议您**将这个项目导入您的IDE**中。
+
+IntelliJ IDEA支持Maven开箱即用，并为Scala开发提供了一个插件。从我们的经验来看，IntelliJ为开发Flink应用程序提供了最好的体验。
+
+对于Eclipse，您需要以下插件，您可以从提供的Eclipse更新站点安装这些插件:
+
+*   _Eclipse 4.x_
+    *   [Scala IDE](http://download.scala-ide.org/sdk/lithium/e44/scala211/stable/site)
+    *   [m2eclipse-scala](http://alchim31.free.fr/m2e-scala/update-site)
+    *   [Build Helper Maven Plugin](https://repo1.maven.org/maven2/.m2e/connectors/m2eclipse-buildhelper/0.15.0/N/0.15.0.201207090124/)
+*   _Eclipse 3.8_
+    *   [Scala IDE for Scala 2.11](http://download.scala-ide.org/sdk/helium/e38/scala211/stable/site) or [Scala IDE for Scala 2.10](http://download.scala-ide.org/sdk/helium/e38/scala210/stable/site)
+    *   [m2eclipse-scala](http://alchim31.free.fr/m2e-scala/update-site)
+    *   [Build Helper Maven Plugin](https://repository.sonatype.org/content/repositories/forge-sites/m2e-extras/0.14.0/N/0.14.0.201109282148/)
+
+### 构建项目
+
+如果您想**构建/打包您的项目**，请转到您的项目目录并运行 ‘`mvn clean package`’ 命令。您将找到一个JAR文件，其中包含您的应用程序，以及您可能作为依赖项添加到应用程序中的连接器和库:`target/&lt;artifact-id&gt;-&lt;version&gt;.jar`。
+
+**注意:** 如果您使用与 _StreamingJob_ 不同的类作为应用程序的主类/入口点，我们建议您更改`pom.xml` 中的`mainClass` 设置。相应的xml的文件。这样，Flink就可以从JAR文件运行应用程序，而无需额外指定主类。
+
+## 下一个步骤
+
+写您的应用程序!
+
+如果您正在编写流应用程序，并且正在寻找编写内容的灵感，请参阅[Stream Processing application Tutorial](//ci.apache.org/projects/flink/flink-docs-release-1.7/tutorials/datastream_api.html#writing-a-flink-program)
+
+如果您正在编写批处理应用程序，并且正在寻找编写的灵感，那么可以参考[Batch Application Examples](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/batch/examples.html)
+
+有关APIa的完整概述，请查看[DataStream API](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/datastream_api.html)和[DataSet API](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/batch/index.html)小节。
+
+[Here](//ci.apache.org/projects/flink/flink-docs-release-1.7/tutorials/local_setup.html)您可以了解如何在本地集群上运行IDE之外的应用程序。
+
+如果您有任何问题，请访问我们的[Mailing List](http://mail-archives.apache.org/mod_mbox/flink-user/)。我们很乐意提供帮助。
+

doc/flink-17-doc-zh/100.md

+ $$ \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\x}{\mathbf{x}} \newcommand{\y}{\mathbf{y}} \newcommand{\wv}{\mathbf{w}} \newcommand{\av}{\mathbf{\alpha}} \newcommand{\bv}{\mathbf{b}} \newcommand{\N}{\mathbb{N}} \newcommand{\id}{\mathbf{I}} \newcommand{\ind}{\mathbf{1}} \newcommand{\0}{\mathbf{0}} \newcommand{\unit}{\mathbf{e}} \newcommand{\one}{\mathbf{1}} \newcommand{\zero}{\mathbf{0}} \newcommand\rfrac[2]{^{#1}\!/_{#2}} \newcommand{\norm}[1]{\left\lVert#1\right\rVert} $$
+
+# 随机异常值选择
+
+## 描述
+
+异常值是一个或多个在数量上偏离大多数数据集的观测值，可能是进一步研究的主题。由Jeroen Janssens[[1]](# Janssens)开发的随机离群点选择(SOS)是一种将一组向量作为输入的无监督随机异常值选择算法。该算法采用基于相似性的离群点选择，并对每个数据点输出一个离群点概率。直观地说，当其他数据点与某个数据点没有足够的关联性时，该数据点就被认为是一个异常值。
+
+异常值检测在许多领域都有应用，例如，日志分析，欺诈检测，噪声消除，新颖性检测，质量控制，传感器监控等。如果传感器出现故障，大概率会输出偏离的值。
+
+有关更多信息，请参阅[Jeroens Janssens博士论文](https://github.com/jeroenjanssens/phd-thesis)关于异常值选择和单类分类的介绍算法。
+
+## 参数
+
+随机异常值选择算法的实现可以通过以下参数控制:
+
+
+| 参数 | 描述 |
+| --- | --- |
+| **Perplexity** | Perplexity可以解释为k-最近邻算法中的k。 与SOS作为邻居的区别不是二元属性，而是概率属性，因此它是实数。 必须介于1和n-1之间，其中n是点数。 通过使用观察数量的平方根可以获得良好的起点. (默认值: **30**) |
+| **ErrorTolerance** | 可接受的误差容限，以在近似亲和力时减少计算时间。 它会牺牲精度以换取减少的计算时间. (默认值: **1e-20**) |
+| **MaxIterations** | 用于近似算法亲和度的最大迭代次数。(默认值: **10**) |
+
+## 例子
+
+
+
+```
+val data = env.fromCollection(List(
+  LabeledVector(0.0, DenseVector(1.0, 1.0)),
+  LabeledVector(1.0, DenseVector(2.0, 1.0)),
+  LabeledVector(2.0, DenseVector(1.0, 2.0)),
+  LabeledVector(3.0, DenseVector(2.0, 2.0)),
+  LabeledVector(4.0, DenseVector(5.0, 8.0)) // The outlier! ))
+
+val sos = new StochasticOutlierSelection().setPerplexity(3)
+
+val outputVector = sos
+  .transform(data)
+  .collect()
+
+val expectedOutputVector = Map(
+  0 -> 0.2790094479202896,
+  1 -> 0.25775014551682535,
+  2 -> 0.22136130977995766,
+  3 -> 0.12707053787018444,
+  4 -> 0.9922779902453757 // The outlier! )
+
+outputVector.foreach(output => expectedOutputVector(output._1) should be(output._2))
+```
+
+
+
+**参考**
+
+[1]J.H.M. Janssens, F. Huszar, E.O. Postma, and H.J. van den Herik. _Stochastic Outlier Selection_. Technical Report TiCC TR 2012-001, Tilburg University, Tilburg, the Netherlands, 2012.
+

doc/flink-17-doc-zh/101.md

+ $$ \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\x}{\mathbf{x}} \newcommand{\y}{\mathbf{y}} \newcommand{\wv}{\mathbf{w}} \newcommand{\av}{\mathbf{\alpha}} \newcommand{\bv}{\mathbf{b}} \newcommand{\N}{\mathbb{N}} \newcommand{\id}{\mathbf{I}} \newcommand{\ind}{\mathbf{1}} \newcommand{\0}{\mathbf{0}} \newcommand{\unit}{\mathbf{e}} \newcommand{\one}{\mathbf{1}} \newcommand{\zero}{\mathbf{0}} \newcommand\rfrac[2]{^{#1}\!/_{#2}} \newcommand{\norm}[1]{\left\lVert#1\right\rVert} $$
+
+# Standard Scaler
+
+## Description
+
+The standard scaler scales the given data set, so that all features will have a user specified mean and variance. In case the user does not provide a specific mean and standard deviation, the standard scaler transforms the features of the input data set to have mean equal to 0 and standard deviation equal to 1. Given a set of input data $x_1, x_2,… x_n$, with mean:
+
+and standard deviation:
+
+The scaled data set $z_1, z_2,…,z_n$ will be:
+
+where $\textit{std}$ and $\textit{mean}$ are the user specified values for the standard deviation and mean.
+
+## Operations
+
+`StandardScaler` is a `Transformer`. As such, it supports the `fit` and `transform` operation.
+
+### Fit
+
+StandardScaler is trained on all subtypes of `Vector` or `LabeledVector`:
+
+*   `fit[T <: Vector]: DataSet[T] => Unit`
+*   `fit: DataSet[LabeledVector] => Unit`
+
+### Transform
+
+StandardScaler transforms all subtypes of `Vector` or `LabeledVector` into the respective type:
+
+*   `transform[T <: Vector]: DataSet[T] => DataSet[T]`
+*   `transform: DataSet[LabeledVector] => DataSet[LabeledVector]`
+
+## Parameters
+
+The standard scaler implementation can be controlled by the following two parameters:
+
+| Parameters | Description |
+| --- | --- |
+| **Mean** | The mean of the scaled data set. (Default value: **0.0**) |
+| **Std** | The standard deviation of the scaled data set. (Default value: **1.0**) |
+
+## Examples
+
+
+
+```
+// Create standard scaler transformer val scaler = StandardScaler()
+.setMean(10.0)
+.setStd(2.0)
+
+// Obtain data set to be scaled val dataSet: DataSet[Vector] = ...
+
+// Learn the mean and standard deviation of the training data scaler.fit(dataSet)
+
+// Scale the provided data set to have mean=10.0 and std=2.0 val scaledDS = scaler.transform(dataSet)
+```
+
+
+

doc/flink-17-doc-zh/102.md

+ $$ \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\x}{\mathbf{x}} \newcommand{\y}{\mathbf{y}} \newcommand{\wv}{\mathbf{w}} \newcommand{\av}{\mathbf{\alpha}} \newcommand{\bv}{\mathbf{b}} \newcommand{\N}{\mathbb{N}} \newcommand{\id}{\mathbf{I}} \newcommand{\ind}{\mathbf{1}} \newcommand{\0}{\mathbf{0}} \newcommand{\unit}{\mathbf{e}} \newcommand{\one}{\mathbf{1}} \newcommand{\zero}{\mathbf{0}} \newcommand\rfrac[2]{^{#1}\!/_{#2}} \newcommand{\norm}[1]{\left\lVert#1\right\rVert} $$
+
+# SVM using CoCoA
+
+## Description
+
+Implements an SVM with soft-margin using the communication-efficient distributed dual coordinate ascent algorithm with hinge-loss function. The algorithm solves the following minimization problem:
+
+with $\mathbf{w}$ being the weight vector, $\lambda$ being the regularization constant,  being the data points and  being the convex loss functions, which can also depend on the labels . In the current implementation the regularizer is the $\ell_2$-norm and the loss functions are the hinge-loss functions:
+
+With these choices, the problem definition is equivalent to a SVM with soft-margin. Thus, the algorithm allows us to train a SVM with soft-margin.
+
+The minimization problem is solved by applying stochastic dual coordinate ascent (SDCA). In order to make the algorithm efficient in a distributed setting, the CoCoA algorithm calculates several iterations of SDCA locally on a data block before merging the local updates into a valid global state. This state is redistributed to the different data partitions where the next round of local SDCA iterations is then executed. The number of outer iterations and local SDCA iterations control the overall network costs, because there is only network communication required for each outer iteration. The local SDCA iterations are embarrassingly parallel once the individual data partitions have been distributed across the cluster.
+
+The implementation of this algorithm is based on the work of [Jaggi et al.](http://arxiv.org/abs/1409.1458)
+
+## Operations
+
+`SVM` is a `Predictor`. As such, it supports the `fit` and `predict` operation.
+
+### Fit
+
+SVM is trained given a set of `LabeledVector`:
+
+*   `fit: DataSet[LabeledVector] => Unit`
+
+### Predict
+
+SVM predicts for all subtypes of FlinkML’s `Vector` the corresponding class label:
+
+*   `predict[T <: Vector]: DataSet[T] => DataSet[(T, Double)]`, where the `(T, Double)` tuple corresponds to (original_features, label)
+
+If we call evaluate with a `DataSet[(Vector, Double)]`, we make a prediction on the class label for each example, and return a `DataSet[(Double, Double)]`. In each tuple the first element is the true value, as was provided from the input `DataSet[(Vector, Double)]` and the second element is the predicted value. You can then use these `(truth, prediction)` tuples to evaluate the algorithm’s performance.
+
+*   `predict: DataSet[(Vector, Double)] => DataSet[(Double, Double)]`
+
+## Parameters
+
+The SVM implementation can be controlled by the following parameters:
+
+| Parameters | Description |
+| --- | --- |
+| **Blocks** | Sets the number of blocks into which the input data will be split. On each block the local stochastic dual coordinate ascent method is executed. This number should be set at least to the degree of parallelism. If no value is specified, then the parallelism of the input DataSet is used as the number of blocks. (Default value: **None**) |
+| **Iterations** | Defines the maximum number of iterations of the outer loop method. In other words, it defines how often the SDCA method is applied to the blocked data. After each iteration, the locally computed weight vector updates have to be reduced to update the global weight vector value. The new weight vector is broadcast to all SDCA tasks at the beginning of each iteration. (Default value: **10**) |
+| **LocalIterations** | Defines the maximum number of SDCA iterations. In other words, it defines how many data points are drawn from each local data block to calculate the stochastic dual coordinate ascent. (Default value: **10**) |
+| **Regularization** | Defines the regularization constant of the SVM algorithm. The higher the value, the smaller will the 2-norm of the weight vector be. In case of a SVM with hinge loss this means that the SVM margin will be wider even though it might contain some false classifications. (Default value: **1.0**) |
+| **Stepsize** | Defines the initial step size for the updates of the weight vector. The larger the step size is, the larger will be the contribution of the weight vector updates to the next weight vector value. The effective scaling of the updates is $\frac{stepsize}{blocks}$. This value has to be tuned in case that the algorithm becomes unstable. (Default value: **1.0**) |
+| **ThresholdValue** | Defines the limiting value for the decision function above which examples are labeled as positive (+1.0). Examples with a decision function value below this value are classified as negative (-1.0). In order to get the raw decision function values you need to indicate it by using the OutputDecisionFunction parameter. (Default value: **0.0**) |
+| **OutputDecisionFunction** | Determines whether the predict and evaluate functions of the SVM should return the distance to the separating hyperplane, or binary class labels. Setting this to true will return the raw distance to the hyperplane for each example. Setting it to false will return the binary class label (+1.0, -1.0) (Default value: **false**) |
+| **Seed** | Defines the seed to initialize the random number generator. The seed directly controls which data points are chosen for the SDCA method. (Default value: **Random Long Integer**) |
+
+## Examples
+
+
+
+```
+import org.apache.flink.api.scala._
+import org.apache.flink.ml.math.Vector
+import org.apache.flink.ml.common.LabeledVector
+import org.apache.flink.ml.classification.SVM
+import org.apache.flink.ml.RichExecutionEnvironment
+
+val pathToTrainingFile: String = ???
+val pathToTestingFile: String = ???
+val env = ExecutionEnvironment.getExecutionEnvironment
+
+// Read the training data set, from a LibSVM formatted file val trainingDS: DataSet[LabeledVector] = env.readLibSVM(pathToTrainingFile)
+
+// Create the SVM learner val svm = SVM()
+  .setBlocks(10)
+
+// Learn the SVM model svm.fit(trainingDS)
+
+// Read the testing data set val testingDS: DataSet[Vector] = env.readLibSVM(pathToTestingFile).map(_.vector)
+
+// Calculate the predictions for the testing data set val predictionDS: DataSet[(Vector, Double)] = svm.predict(testingDS)
+```
+
+
+

doc/flink-17-doc-zh/103.md

+
+
+# 最佳实践
+
+这里是Flink常见问题的解决办法
+
+## 传递命令行参数并在Flink程序中使用
+
+大部分Flink程序（无论是批处理还是流式计算）都依赖于外部的参数配置. 比如指定输入输出位置(路径或地址)，系统参数(并行度和运行时配置)，程序相关的参数配置。
+
+Flink提供一个叫 `ParameterTool` 的工具类来解决此问题。 `ParameterTool` 并不是解决这个问题的唯一办法，其他框架，比如 [Commons CLI](https://commons.apache.org/proper/commons-cli/) 和 [argparse4j](http://argparse4j.sourceforge.net/) 也可以解决这个问题.
+
+### 通过 `ParameterTool` 来获取配置值
+
+`ParameterTool` 工具提供了很多读取配置的静态方法， 工具类内部实现类似于 `Map<String, String>`, 所以和项目代码风格不会有冲突
+
+#### 从 `.properties` 文件获取配置
+
+下面是个读 [Properties](https://docs.oracle.com/javase/tutorial/essential/environment/properties.html) 文件并将内容转换成key-value的例子
+
+
+
+```
+String propertiesFilePath = "/home/sam/flink/myjob.properties";
+ParameterTool parameter = ParameterTool.fromPropertiesFile(propertiesFilePath);
+
+File propertiesFile = new File(propertiesFilePath);
+ParameterTool parameter = ParameterTool.fromPropertiesFile(propertiesFile);
+
+InputStream propertiesFileInputStream = new FileInputStream(file);
+ParameterTool parameter = ParameterTool.fromPropertiesFile(propertiesFileInputStream);
+```
+
+
+
+#### 从命令行中获取配置
+
+下面是命令行参数的例子，例如在命令行后加入 `--input hdfs:///mydata --elements 42` 参数。
+
+
+
+```
+public static void main(String[] args) {
+    ParameterTool parameter = ParameterTool.fromArgs(args);
+    // .. regular code ..
+```
+
+
+
+#### 从系统属性中获取配置
+
+启动JVM时，可以把系统属性传递给程序，例如: `-Dinput=hdfs:///mydata`。 可以通过 `ParameterTool` 来获取配置，例如:
+
+
+
+```
+ParameterTool parameter = ParameterTool.fromSystemProperties();
+```
+
+
+
+### 在Flink程序中使用配置参数
+
+根据上文，我们已经获取了参数，下面是如何使用它们。
+
+**使用 `ParameterTool` 工具类**
+
+ `ParameterTool` 工具类有直接访问值的方法，例如：
+
+
+
+```
+ParameterTool parameters = // ...
+parameter.getRequired("input");
+parameter.get("output", "myDefaultValue");
+parameter.getLong("expectedCount", -1L);
+parameter.getNumberOfParameters()
+// .. there are more methods available.
+```
+
+
+
+您可以直接在 `main()` 中使用， 您可以通过如下代码来设置算子的并行度:
+
+
+
+```
+ParameterTool parameters = ParameterTool.fromArgs(args);
+int parallelism = parameters.get("mapParallelism", 2);
+DataSet<Tuple2<String, Integer>> counts = text.flatMap(new Tokenizer()).setParallelism(parallelism);
+```
+
+
+
+因为 `ParameterTool` 是可序列化的,所以你可以把它作为函数的参数:
+
+
+
+```
+ParameterTool parameters = ParameterTool.fromArgs(args);
+DataSet<Tuple2<String, Integer>> counts = text.flatMap(new Tokenizer(parameters));
+```
+
+
+
+然后在函数内通过上文方法获取值.
+
+#### 注册全局参数
+
+通过 `ExecutionConfig` 注册为全局参数后，可以被JobManager Web界面中的配置值和代码内的所有函数来访问.
+
+注册为全局参数:
+
+
+
+```
+ParameterTool parameters = ParameterTool.fromArgs(args);
+
+// set up the execution environment
+final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
+env.getConfig().setGlobalJobParameters(parameters);
+```
+
+
+
+在代码中的函数内访问:
+
+
+
+```
+public static final class Tokenizer extends RichFlatMapFunction<String, Tuple2<String, Integer>> {
+
+    @Override
+    public void flatMap(String value, Collector<Tuple2<String, Integer>> out) {
+	ParameterTool parameters = (ParameterTool)
+	    getRuntimeContext().getExecutionConfig().getGlobalJobParameters();
+	parameters.getRequired("input");
+	// .. do more ..
+```
+
+
+
+## 超大TupleX类型命令
+
+强烈推荐使用 POJO 来代替有很多字段的 `TupleX`, POJO 可以代替多字段的 `Tuple`类型.
+
+**Example**
+
+而不是使用:
+
+
+
+```
+Tuple11<String, String, ..., String> var = new ...;
+```
+
+
+
+把大型元组类型继承为自定义类型也会方便很多
+
+
+```
+CustomType var = new ...;
+
+public static class CustomType extends Tuple11<String, String, ..., String> {
+    // constructor matching super
+}
+```
+
+
+
+## 用Logback而不是Log4j
+
+**注意：本教程适用于Flink 0.10及其以上版本**
+
+Apache Flink 使用 [slf4j](http://www.slf4j.org/) 作为记录日志的抽象.，建议用户在使用事采用sfl4j来记录日志。
+
+Sfl4j 是一个日志记录的抽象接口，可以在代码中使用不同的日志实现，例如 [log4j](http://logging.apache.org/log4j/2.x/) 或者 [Logback](http://logback.qos.ch/)。
+
+Flink 默认依赖于Log4j. 本页介绍如何使用在Flink中使用Logback. 用户报告说， 他们可以通过 Graylog来建立一个集中式日志收集处理系统.
+
+以下代码可以获取Logger实例:
+
+
+
+```
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class MyClass implements MapFunction {
+    private static final Logger LOG = LoggerFactory.getLogger(MyClass.class);
+    // ...
+```
+
+
+
+### 当通过 IDE 或者 Java程序运行Flink程序时如何使用Logback
+
+在任何情况下，类都是通过依赖管理器（例如maven）创建的路径来执行, Flink 会把log4j的相关依赖拉入到classpath中.
+
+因此, 需要排除掉Flink的log4j依赖. 下面的pom文件从  [Flink quickstart](./projectsetup/java_api_quickstart.html)创建的maven项目.
+
+ `pom.xml`文件需要进行如下更改:
+
+
+
+```
+<dependencies>
+	<!-- Add the two required logback dependencies -->
+	<dependency>
+		<groupId>ch.qos.logback</groupId>
+		<artifactId>logback-core</artifactId>
+		<version>1.1.3</version>
+	</dependency>
+	<dependency>
+		<groupId>ch.qos.logback</groupId>
+		<artifactId>logback-classic</artifactId>
+		<version>1.1.3</version>
+	</dependency>
+
+	<!-- Add the log4j -> sfl4j (-> logback) bridge into the classpath
+	 Hadoop is logging to log4j! -->
+	<dependency>
+		<groupId>org.slf4j</groupId>
+		<artifactId>log4j-over-slf4j</artifactId>
+		<version>1.7.7</version>
+	</dependency>
+
+	<dependency>
+		<groupId>org.apache.flink</groupId>
+		<artifactId>flink-java</artifactId>
+		<version>1.7.1</version>
+		<exclusions>
+			<exclusion>
+				<groupId>log4j</groupId>
+				<artifactId>*</artifactId>
+			</exclusion>
+			<exclusion>
+				<groupId>org.slf4j</groupId>
+				<artifactId>slf4j-log4j12</artifactId>
+			</exclusion>
+		</exclusions>
+	</dependency>
+	<dependency>
+		<groupId>org.apache.flink</groupId>
+		<artifactId>flink-streaming-java_2.11</artifactId>
+		<version>1.7.1</version>
+		<exclusions>
+			<exclusion>
+				<groupId>log4j</groupId>
+				<artifactId>*</artifactId>
+			</exclusion>
+			<exclusion>
+				<groupId>org.slf4j</groupId>
+				<artifactId>slf4j-log4j12</artifactId>
+			</exclusion>
+		</exclusions>
+	</dependency>
+	<dependency>
+		<groupId>org.apache.flink</groupId>
+		<artifactId>flink-clients_2.11</artifactId>
+		<version>1.7.1</version>
+		<exclusions>
+			<exclusion>
+				<groupId>log4j</groupId>
+				<artifactId>*</artifactId>
+			</exclusion>
+			<exclusion>
+				<groupId>org.slf4j</groupId>
+				<artifactId>slf4j-log4j12</artifactId>
+			</exclusion>
+		</exclusions>
+	</dependency>
+</dependencies>
+```
+
+
+
+ `&lt;dependencies&gt;` 部分进行了以下更改:
+
+*   从Flink中排除了所有的 `log4j` 依赖项: maven会忽略所有Flink对log4j的依赖传递。
+*   从Flink中排除了所有的 `slf4j-log4j12` 依赖项: 因为我们要使用sl4j来进行logback绑定, 所以我们必须将slf4j从log4j中的绑定删除。
+*   增加 `logback-core` 和 `logback-classic`的依赖项。
+*   增加 `log4j-over-slf4j`依赖项。 `log4j-over-slf4j` 是一个可以通过Log4j API来使用Slf4j接口的工具。 Flink依赖于Hadoop，然而Hadoop直接使用Log4j进行日志记录。 因此, 我们需要将所有日志记录器调用从Log4j重定向到Slf4j，后者又记录到Logback。
+
+请注意，您需要对新添加到pom中的Flink依赖都进行手动排除。
+
+您还需要检查其他非Flink依赖项是不是引入了log4j绑定，可以通过`mvn dependency:tree`来对项目依赖进行分析.
+
+### 在集群上的Flink程序如何使用Logback
+
+This tutorial is applicable when running Flink on YARN or as a standalone cluster.
+本教程适用于在YARN上运行Flink或其他独立集群。
+
+为了在Flink中使用Logback而不是Log4j, 您首先需要从 `lib/` 目录移除 `log4j-1.2.xx.jar` 和 `sfl4j-log4j12-xxx.jar`。
+
+然后, 您需要把以下Jar文件放到 `lib/` 目录下:
+
+*   `logback-classic.jar`
+*   `logback-core.jar`
+*   `log4j-over-slf4j.jar`: 此桥接器需要存在于classpath中，以便Hadoop将日志记录器调用从Log4j重定向到Slf4j。
+
+请注意，此情况你需要在向YARN提交Flink作业时显式指定 `lib/` 目录。 例如:
+
+
+`./bin/flink run -yt $FLINK_HOME/lib &lt;... remaining arguments ...&gt;`
+
+

doc/flink-17-doc-zh/104.md

+
+
+# API 迁移指南
+
+## 从Flink 1.3+ 到 Flink 1.7
+
+### TypeSerializer 的变化
+
+这部分主要与实现`TypeSerializer`接口来自定义序列化的用户有关。
+
+原来的 `TypeSerializerConfigSnapshot` 抽象接口被弃用了, 并且将在将来完全删除，取而代之的是新的 `TypeSerializerSnapshot`. 详情请参考 [Migrating from deprecated serializer snapshot APIs before Flink 1.7](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/state/custom_serialization.html#migration-from-deprecated-serializer-snapshot-apis-before-Flink-1.7)。
+
+## 从 Flink 1.2 迁移到 Flink 1.3
+
+自Flink 1.2以来，有一些API已被更改。大多数更改都记录在其特定文档中。以下是API更改的综合列表以及升级到Flink 1.3时迁移详细信息的链接。
+
+### `TypeSerializer` 接口变化
+
+这主要适用于自定义 `TypeSerializer`接口的用户
+
+从Flink 1.3开始，添加了两个与保存点恢复的串行器兼容性相关的其他方法。 有关如何实现这些方法的更多详细信息，请参阅 [序列化升级和兼容性](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/state/custom_serialization.html#handling-serializer-upgrades-and-compatibility)
+
+### `ProcessFunction` 是 `RichFunction`
+
+从Flink 1.2, `ProcessFunction` 引入，并有了多种实现例如 `RichProcessFunction` 。 从Flink 1.3,开始`RichProcessFunction` 被移除了， 现在 `ProcessFunction` 始终是 `RichFunction` 并且可以访问运行时上下文。
+
+### Flink CEP 库API更改
+
+Flink 1.3中的CEP库新增了许多新函数，请参阅 [CEP迁移文档](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/libs/cep.html#migrating-from-an-older-flink-version) 。
+
+### Flink core 中移除了Logger的依赖
+
+Flink1.3以后，用户可以选用自己期望的日志框架了，Flink移除了日志记录框架的依赖。
+
+实例和快速入门的demo已经指定了日志记录器，不会有问题，其他项目，请确保添加日志依赖，比如Maven的 `pom.xml`，中需要增加
+
+
+
+```
+<dependency>
+    <groupId>org.slf4j</groupId>
+    <artifactId>slf4j-log4j12</artifactId>
+    <version>1.7.7</version>
+</dependency>
+
+<dependency>
+    <groupId>log4j</groupId>
+    <artifactId>log4j</artifactId>
+    <version>1.2.17</version>
+</dependency>
+```
+
+
+
+## 从 Flink 1.1 到 Flink 1.2的迁移
+
+正如 [状态文档](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/state/state.html)中所说，Flink 有两种状态: **keyed** 和 **non-keyed** 状态 (也被称作 **operator** 状态). 这两种类型都可用于 算子和用户定义的函数。文档将指导您完成从Flink 1.1函数代码迁移到Flink 1.2的过程，并介绍Flink 1.2中引入的一些重要内部更改，这些改变涉及到Flink 1.1中对齐窗口操作的弃用。 (请参阅 时间对齐窗口算子[Aligned Processing Time Window Operators](#aligned-processing-time-window-operators)).
+
+迁移有两个目标：
+
+1.  引入Flink1.2中引入的新函数，比如自适应（rescaling）
+
+2.  确保新Flink 1.2作业能够从Flink 1.1的保存点恢复执行
+
+按照本指南操作可以把正在运行的Flink1.1作业迁移到Flink1.2中。前提需要在Flink1.1中使用 [保存点](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/state/savepoints.html) 并把保存点作为Flink1.2作业的起点。这样，Flink1.2就可以从之前中断的位置恢复执行了。
+
+### 用户函数示例
+
+本文档其余部分使用 `CountMapper` 和 `BufferingSink` 函数作为示例。第一个函数是 **keyed** 状态，第二个不是 s **non-keyed** 状态，Flink 1.1中上述两个函数的代码如下：
+
+
+
+```
+public class CountMapper extends RichFlatMapFunction<Tuple2<String, Integer>, Tuple2<String, Integer>> {
+
+    private transient ValueState<Integer> counter;
+
+    private final int numberElements;
+
+    public CountMapper(int numberElements) {
+        this.numberElements = numberElements;
+    }
+
+    @Override
+    public void open(Configuration parameters) throws Exception {
+        counter = getRuntimeContext().getState(
+            new ValueStateDescriptor<>("counter", Integer.class, 0));
+    }
+
+    @Override
+    public void flatMap(Tuple2<String, Integer> value, Collector<Tuple2<String, Integer>> out) throws Exception {
+        int count = counter.value() + 1;
+        counter.update(count);
+
+        if (count % numberElements == 0) {
+            out.collect(Tuple2.of(value.f0, count));
+            counter.update(0); // reset to 0
+        }
+    }
+}
+
+public class BufferingSink implements SinkFunction<Tuple2<String, Integer>>,
+    Checkpointed<ArrayList<Tuple2<String, Integer>>> {
+
+    private final int threshold;
+
+    private ArrayList<Tuple2<String, Integer>> bufferedElements;
+
+    BufferingSink(int threshold) {
+        this.threshold = threshold;
+        this.bufferedElements = new ArrayList<>();
+    }
+
+    @Override
+    public void invoke(Tuple2<String, Integer> value) throws Exception {
+        bufferedElements.add(value);
+        if (bufferedElements.size() == threshold) {
+            for (Tuple2<String, Integer> element: bufferedElements) {
+                // send it to the sink
+            }
+            bufferedElements.clear();
+        }
+    }
+
+    @Override
+    public ArrayList<Tuple2<String, Integer>> snapshotState(
+        long checkpointId, long checkpointTimestamp) throws Exception {
+        return bufferedElements;
+    }
+
+    @Override
+    public void restoreState(ArrayList<Tuple2<String, Integer>> state) throws Exception {
+        bufferedElements.addAll(state);
+    }
+}
+```
+
+
+
+`CountMapper` 是一个按表格分组输入`(word, 1)`的 `RichFlatMapFunction` ， 函数为每个传入的key保存一个计数器 (`ValueState&lt;Integer&gt; counter`) 并且 ，如果某个单词的出现次数超过用户提供的阈值，则会发出一个包含单词本身和出现次数的元组。
+
+ `BufferingSink` 是一个 `SinkFunction` 接收方 ( `CountMapper`可能的输出) ，直到达到用户定义的最终状态之前会一直缓存数据，这可以避免频繁的对数据库或者是存储系统的操作，通常这些操作都是比较耗时或开销比较大的。为了以容错方式进行缓冲，缓冲数据元保存在列表(`bufferedElements`) 列表会被定期被检查点保存。
+
+### 状态 API 迁移
+
+要使用Flink 1.2的新函数，应修改上面的代码来完成新的状态抽象。完成这些更改后，就可以实现作业的并行度的修改（向上或向下扩展），并确保新版本的作业将从之前作业停止的位置开始执行。
+
+**Keyed State:** 需要注意的是，如果代码中只有**keyed state**，那么Flink1.1的代码也适用于1.2版本，并且完全支持新函数和向下兼容。可以仅针代码格式进行更改，但这只是风格/习惯问题。
+
+综上所述，本章我们重点阐述 **non-keyed state**的迁移
+
+#### 自适应和新状态抽象
+
+第一个修改是 `Checkpointed&lt;T extends Serializable&gt;` 接口有了新的实现。在 Flink 1.2中，有状态函数可以实现更通用的 `CheckpointedFunction` 接口或 `ListCheckpointed&lt;T extends Serializable&gt;` 接口 ，和之前版本的 `Checkpointed` 类似。
+
+在这两种情况中，非键合状态预期是一个 _可序列化_ 的 `List` ，对象彼此独立，这样可以在自适应的时候重新分配，意味着， 这些对象是可以重新分区非被Keys化状态的最细粒度。例如，如果并行度为1的  `BufferingSink` 有 `(test1, 2)` 和 `(test2, 2)`两个数据，当并行度增加到2时， `(test1, 2)` 可能在task 0中，而 `(test2, 2)` 可能在task 1中。
+
+更详细的信息可以参阅[状态文档](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/state/index.html).
+
+##### ListCheckpointed
+
+ `ListCheckpointed` 接口需要实现两个方法：
+
+
+
+```
+List<T> snapshotState(long checkpointId, long timestamp) throws Exception;
+
+void restoreState(List<T> state) throws Exception;
+```
+
+
+
+它的语义和之前的 `Checkpointed` 接口类似，唯一区别是，现在 `snapshotState()` 返回的是检查点对象列表， 如前所述， `restoreState` 必须在恢复的时候，处理这个列表。如果状态不是重新分区，可以随时返回 `Collections.singletonList(MY_STATE)` 的 `snapshotState()`。 更新的代码 `BufferingSink` 如下：
+
+
+
+```
+public class BufferingSinkListCheckpointed implements
+        SinkFunction<Tuple2<String, Integer>>,
+        ListCheckpointed<Tuple2<String, Integer>>,
+        CheckpointedRestoring<ArrayList<Tuple2<String, Integer>>> {
+
+    private final int threshold;
+
+    private transient ListState<Tuple2<String, Integer>> checkpointedState;
+
+    private List<Tuple2<String, Integer>> bufferedElements;
+
+    public BufferingSinkListCheckpointed(int threshold) {
+        this.threshold = threshold;
+        this.bufferedElements = new ArrayList<>();
+    }
+
+    @Override
+    public void invoke(Tuple2<String, Integer> value) throws Exception {
+        this.bufferedElements.add(value);
+        if (bufferedElements.size() == threshold) {
+            for (Tuple2<String, Integer> element: bufferedElements) {
+                // send it to the sink
+            }
+            bufferedElements.clear();
+        }
+    }
+
+    @Override
+    public List<Tuple2<String, Integer>> snapshotState(
+            long checkpointId, long timestamp) throws Exception {
+        return this.bufferedElements;
+    }
+
+    @Override
+    public void restoreState(List<Tuple2<String, Integer>> state) throws Exception {
+        if (!state.isEmpty()) {
+            this.bufferedElements.addAll(state);
+        }
+    }
+
+    @Override
+    public void restoreState(ArrayList<Tuple2<String, Integer>> state) throws Exception {
+        // this is from the CheckpointedRestoring interface.
+        this.bufferedElements.addAll(state);
+    }
+}
+```
+
+
+
+更新后的函数也实现了 `CheckpointedRestoring` 接口。这是出于向后兼容性原因，更多细节将在本节末尾解释。
+
+##### CheckpointedFunction
+
+`CheckpointedFunction` 接口也需要实现这两个方法。
+
+
+
+```
+void snapshotState(FunctionSnapshotContext context) throws Exception;
+
+void initializeState(FunctionInitializationContext context) throws Exception;
+```
+
+
+
+在Flink 1.1中， 检查点执行会调用`snapshotState()` 方法，但是现在当用户每次初始化自定义函数时，会调用 `initializeState()` (对应 `restoreState()`) ，而不是在恢复的情况下调用，鉴于此， `initializeState()` 不仅是初始化不同类型状态的地方，而且还包括状态恢复逻辑。实现了 `CheckpointedFunction` 接口的 `BufferingSink` 代码如下所示
+
+
+
+```
+public class BufferingSink implements SinkFunction<Tuple2<String, Integer>>,
+        CheckpointedFunction, CheckpointedRestoring<ArrayList<Tuple2<String, Integer>>> {
+
+    private final int threshold;
+
+    private transient ListState<Tuple2<String, Integer>> checkpointedState;
+
+    private List<Tuple2<String, Integer>> bufferedElements;
+
+    public BufferingSink(int threshold) {
+        this.threshold = threshold;
+        this.bufferedElements = new ArrayList<>();
+    }
+
+    @Override
+    public void invoke(Tuple2<String, Integer> value) throws Exception {
+        bufferedElements.add(value);
+        if (bufferedElements.size() == threshold) {
+            for (Tuple2<String, Integer> element: bufferedElements) {
+                // send it to the sink
+            }
+            bufferedElements.clear();
+        }
+    }
+
+    @Override
+    public void snapshotState(FunctionSnapshotContext context) throws Exception {
+        checkpointedState.clear();
+        for (Tuple2<String, Integer> element : bufferedElements) {
+            checkpointedState.add(element);
+        }
+    }
+
+    @Override
+    public void initializeState(FunctionInitializationContext context) throws Exception {
+        checkpointedState = context.getOperatorStateStore().
+            getSerializableListState("buffered-elements");
+
+        if (context.isRestored()) {
+            for (Tuple2<String, Integer> element : checkpointedState.get()) {
+                bufferedElements.add(element);
+            }
+        }
+    }
+
+    @Override
+    public void restoreState(ArrayList<Tuple2<String, Integer>> state) throws Exception {
+        // this is from the CheckpointedRestoring interface.
+        this.bufferedElements.addAll(state);
+    }
+}
+```
+
+
+
+ `initializeState` 方法是需要传入 `FunctionInitializationContext`，用于初始化non-keyed 状态的 “容器”，容器的类型是 `ListState`，供 non-keyed 状态的对象被检查点存储时使用:
+
+`this.checkpointedState = context.getOperatorStateStore().getSerializableListState("buffered-elements");`
+
+初始化之后，调用 `isRestored()` 方法可以获取当前是否在恢复。如果是 `true`, 表示正在恢复。
+
+正如下面的代码所示，在状态初始化期间恢复 `BufferingSink`， `ListState` 中保存的变量可以被 `snapshotState()`使用，  `ListState` 会清除掉之前检查点存储的对象，然后存储当前检查点的对象。
+
+当然， keyed 状态也可以在 `initializeState()` 方法中初始化， 可以使用 `FunctionInitializationContext` 来完成初始化，而不是使用Flink1.1中的 `RuntimeContext`，如果`CheckpointedFunction` 要在 `CountMapper` 中使用该接口，则可以不使用 `open()` 方法， `snapshotState()` 和 `initializeState()` 方法如下所示:
+
+
+
+```
+public class CountMapper extends RichFlatMapFunction<Tuple2<String, Integer>, Tuple2<String, Integer>>
+        implements CheckpointedFunction {
+
+    private transient ValueState<Integer> counter;
+
+    private final int numberElements;
+
+    public CountMapper(int numberElements) {
+        this.numberElements = numberElements;
+    }
+
+    @Override
+    public void flatMap(Tuple2<String, Integer> value, Collector<Tuple2<String, Integer>> out) throws Exception {
+        int count = counter.value() + 1;
+        counter.update(count);
+
+        if (count % numberElements == 0) {
+            out.collect(Tuple2.of(value.f0, count));
+            counter.update(0); // reset to 0
+        }
+    }
+
+    @Override
+    public void snapshotState(FunctionSnapshotContext context) throws Exception {
+        // all managed, nothing to do.
+    }
+
+    @Override
+    public void initializeState(FunctionInitializationContext context) throws Exception {
+        counter = context.getKeyedStateStore().getState(
+            new ValueStateDescriptor<>("counter", Integer.class, 0));
+    }
+}
+```
+
+
+
+请注意， `snapshotState()` 方法为空，因为Flink本身负责在检查点时就会存储Keys化的对象
+
+#### 向后兼容Flink 1.1
+
+到目前为止，我们已经了解如何修改函数来引入Flink 1.2的新函数。剩下的问题是“我可以确保我的修改后的（Flink 1.2）作业将从我从Flink 1.1运行的作业停止的位置开始吗？”。
+
+答案是肯定的，而这样做的方式非常简单。对于被Keys化状态，什么都不需要做。Flink将负责从Flink 1.1恢复状态。对于非被Keys化状态，新函数必须像上面代码一样实现 `CheckpointedRestoring` 接口，还有个办法，需要熟悉Flink1.1的 `restoreState()` 和`Checkpointed` 接口，然后修改 `BufferingSink`, `restoreState()` 方法完成和之前一样的功能。
+
+### 时间对齐窗口算子
+
+在Flink 1.1中，只有在没有指定的触发器的时， `timeWindow()` 才会实例化特殊的数据类型 `WindowOperator`。 它可以是 `AggregatingProcessingTimeWindowOperator` 或 `AccumulatingProcessingTimeWindowOperator`.。这两个算子都可以称之为时间对齐窗口算子，因为它们假定输入数据是按顺序到达，当在处理时间中操作时，这是有效的，元素到达窗口操作时获得时间。算子仅限于使用内存状态，并且优化了用于存储数据的元素结构。
+
+在Flink 1.2中，不推荐使用对齐窗口算子，并且所有窗口算子操作都通过泛型`WindowOperator`来实现。迁移不需要更改Flink 1.1作业的代码，因为Flink将读取Flink 1.1保存点中对齐的窗口 算子存储的状态，将其转换为与泛型相兼容的格式 `WindowOperator`，并使用通用的 `WindowOperator`。
+
+虽然是已经弃用这个方法，但是在Flink 1.2 中仍然是可以使用的，通过特殊的 `WindowAssigners` 可以实现这个目的。 `SlidingAlignedProcessingTimeWindows` 和 `TumblingAlignedProcessingTimeWindows` assigners，分别是滑动窗口和滚动窗口，使用对齐窗口的Flink 1.2作业必须是一项新作业，因为在使用这些 算子时无法从Flink 1.1保存点恢复执行。
+
+注意时间对齐窗口算子**不提供自适应** 而且 **不向下兼容** Flink 1.1.
+
+在Flink 1.2中使用对齐窗口 算子的代码如下所示：
+
+
+
+```
+// for tumbling windows
+DataStream<Tuple2<String, Integer>> window1 = source
+    .keyBy(0)
+    .window(TumblingAlignedProcessingTimeWindows.of(Time.of(1000, TimeUnit.MILLISECONDS)))
+    .apply(your-function)
+
+// for sliding windows
+DataStream<Tuple2<String, Integer>> window1 = source
+    .keyBy(0)
+    .window(SlidingAlignedProcessingTimeWindows.of(Time.seconds(1), Time.milliseconds(100)))
+    .apply(your-function)
+```
+
+
+
+
+
+```
+// for tumbling windows val window1 = source
+    .keyBy(0)
+    .window(TumblingAlignedProcessingTimeWindows.of(Time.of(1000, TimeUnit.MILLISECONDS)))
+    .apply(your-function)
+
+// for sliding windows val window2 = source
+    .keyBy(0)
+    .window(SlidingAlignedProcessingTimeWindows.of(Time.seconds(1), Time.milliseconds(100)))
+    .apply(your-function)
+```
+
+
+

doc/flink-17-doc-zh/105.md

+
+
+# 独立集群
+
+这里给出如何在集群上以完全分布式方式运行Flink程序的说明 .
+
+## 需求
+
+### 软件需求
+
+Flink 需要运行在类unix环境下,比如 **Linux**, **Mac OS X**, 和 **Cygwin** (适用于Windows) 希望集群中至少由一个**主节点**和最少一个**工作节点**组成。  在开始运行前，需要保证 **每个节点**上都安装了一下软件:
+
+*   **Java 1.8.x** 及其以上版本,
+*   **ssh** (必须运行sshd才能使用管理远程组件的Flink脚本)
+
+如果您的群集不满足这些软件要求，则需要安装/升级软件来达到要求.
+
+通过 **SSH互信** 和 **相同的目录结构** 这样我们的脚本就能控制所有内容.
+
+### `JAVA_HOME` 配置
+
+Flink程序要求所有节点上必须设置`JAVA_HOME`环境变量，并`JAVA_HOME`环境变量指向JAVA的安装目录.
+
+可以通过 `conf/flink-conf.yaml` 文件中的 `env.java.home` 配置来设置配置.
+
+## Flink 安装
+
+转到 [下载页](http://flink.apache.org/downloads.html) 下载Flink安装程序. 程序中如果需要使用Hadoop，则选择的Flink版本要保证和 **Hadoop 版本**相匹配。否则可以选择任何版本
+
+下载完毕后，复制到主节点并解压:
+
+```
+tar xzf flink-*.tgz
+cd flink-*
+```
+
+
+
+### 配置 Flink
+
+解压缩之后，需要修改Flink的配置文件，配置文件名称为 _conf/flink-conf.yaml_。
+
+修改 `jobmanager.rpc.address` 配置值为主节点，设置 `jobmanager.heap.mb` 和 `taskmanager.heap.mb` 用来指定JVM在每个节点上分配的最大堆内存量。
+
+这些设置值单位为 MB. 如果某工作节点有更多主内存可以分配给Flink系统，可以修改节点上的 `FLINK_TM_HEAP` 配置来覆盖默认值。
+
+最后，必须提供集群中所有节点的列表，这些节点将用作工作节点。与HDFS配置类似, 编辑_conf/slaves_文件，并输入每个工作节点的IP /主机名. 每个工作节点将运行 _TaskManager_.
+
+下图说明了具有三个节点 （IP 地址 从_10.0.0.1_ 到 _10.0.0.3_ 和对应的主机名 _master_, _worker1_, _worker2_）的集群配置， 并显示了配置文件的内容（需要在所有计算机上的相同路径上访问））:
+
+![](img/quickstart_cluster.png)
+
+/path/to/**flink/conf/
+flink-conf.yaml**
+
+```
+jobmanager.rpc.address: 10.0.0.1
+```
+
+/path/to/**flink/
+conf/slaves**
+
+```
+10.0.0.2
+10.0.0.3
+```
+
+Flink目录必须每个worker上同一路径下都可用。可以使用共享NFS目录，也可以将整个Flink目录复制到每个工作节点。
+
+有关详细信息和其他配置选项，请参阅 [configuration page](../config.html) 。
+
+需要额外注意的一些配置是，
+
+*   每个JobManager的可用内存总量 (`jobmanager.heap.mb`),
+*   每个TaskManager的可用内容总量 (`taskmanager.heap.mb`),
+*   每台机器的可用CPU数量 (`taskmanager.numberOfTaskSlots`),
+*   集群中的CPU总数 (`parallelism.default`)
+*   临时目录 (`taskmanager.tmp.dirs`)
+
+这些是对Flink非常重要的配置值。
+
+### 启动Flink
+
+以下脚本在本地节点上启动JobManager，并通过ssh连接到所有_slaves_文件中定义的所有工作节点，启动TaskManger。 现在Flink系统已启动并正在运行。  在本地运行的JobManager 将通过配置的RPC端口来接受作业提交。
+
+A假设您在主节点上并在Flink目录中:
+
+```
+bin/start-cluster.sh
+```
+
+
+
+停止Flink对应的脚本是 `stop-cluster.sh`。
+
+### 将JobManager/TaskManager实例添加到群集
+
+可以使用 `bin/jobmanager.sh` 和 `bin/taskmanager.sh` 脚本将JobManager和TaskManager实例添加到正在运行的集群中.
+
+#### 添加 JobManager
+
+
+
+```
+bin/jobmanager.sh ((start|start-foreground) [host] [webui-port])|stop|stop-all
+```
+
+
+
+#### 添加 TaskManager
+
+
+
+```
+bin/taskmanager.sh start|start-foreground|stop|stop-all
+```
+
+
+
+需要保证在对应的机器上调用启动/停止脚本。
+

doc/flink-17-doc-zh/106.md

+
+
+# YARN 配置
+
+## 快速开始
+
+### 启动YARN
+
+启动一个有 4 Task Managers (每个4 GB堆内存)的YARN session命令:
+
+
+
+```
+# get the hadoop2 package from the Flink download page at
+# http://flink.apache.org/downloads.html
+curl -O <flink_hadoop2_download_url>
+tar xvzf flink-1.7.1-bin-hadoop2.tgz
+cd flink-1.7.1/
+./bin/yarn-session.sh -n 4 -jm 1024m -tm 4096m
+```
+
+
+
+ `-s`参数可以指定每个TaskManager的处理槽数。建议将插槽数设置为每台计算机的处理器数。
+
+YARN启动后，可以通过 `./bin/flink` 工具来提交Flink作业.
+
+### 在YARN上运行Flink作业
+
+
+
+```
+# get the hadoop2 package from the Flink download page at
+# http://flink.apache.org/downloads.html
+curl -O <flink_hadoop2_download_url>
+tar xvzf flink-1.7.1-bin-hadoop2.tgz
+cd flink-1.7.1/
+./bin/flink run -m yarn-cluster -yn 4 -yjm 1024m -ytm 4096m ./examples/batch/WordCount.jar
+```
+
+
+
+## Flink YARN Session
+
+Apache [Hadoop YARN](http://hadoop.apache.org/) 是一个集群资源管理框架。它允许在群集上运行各种分布式应用程序。Flink在YARN上运行。如果已经有YARN，用户不必安装其他任何东西。
+
+**系统配置需求**
+
+*   版本不低于Apache Hadoop 2.2
+*   HDFS（Hadoop分布式文件系统）（或Hadoop支持的其他分布式文件系统）
+
+如果在使用Flink YARN时遇到问题，请查阅 [FAQ section](http://flink.apache.org/faq.html#yarn-deployment).
+
+### 启动 Flink 会话
+
+以下说明了解如何在YARN群集中启动Flink会话。
+
+会话将启动Flink程序运行所必须的服务 (JobManager 和 TaskManagers) 这样就可以把程序提交到集群中.需要注意的是，可以在会话中运行多个程序
+
+#### Download Flink
+
+从 [下载页](http://flink.apache.org/downloads.html)下载Hadoop版本>2的安装包. 它包含所需的文件。
+
+通过下面命令解压:
+
+
+
+```
+tar xvzf flink-1.7.1-bin-hadoop2.tgz
+cd flink-1.7.1/
+```
+
+
+
+#### 启动一个会话
+
+通过以下命令启动一个session
+
+
+
+```
+./bin/yarn-session.sh
+```
+
+
+
+命令将显示以下概述：
+
+
+
+```
+Usage:
+   Required
+     -n,--container <arg>   Number of YARN container to allocate (=Number of Task Managers)
+   Optional
+     -D <arg>                        Dynamic properties
+     -d,--detached                   Start detached
+     -jm,--jobManagerMemory <arg>    Memory for JobManager Container with optional unit (default: MB)
+     -nm,--name                      Set a custom name for the application on YARN
+     -q,--query                      Display available YARN resources (memory, cores)
+     -qu,--queue <arg>               Specify YARN queue.
+     -s,--slots <arg>                Number of slots per TaskManager
+     -tm,--taskManagerMemory <arg>   Memory per TaskManager Container with optional unit (default: MB)
+     -z,--zookeeperNamespace <arg>   Namespace to create the Zookeeper sub-paths for HA mode
+```
+
+
+
+需要注意的是，客户端需要 `YARN_CONF_DIR` 或 `HADOOP_CONF_DIR` 环境变量来保证可以读取 YARN 和 HDFS 配置.
+
+**举例:** 以下命令可以分配10个TaskManager，每个TaskManager有8 GB内存和32个处理插槽
+
+
+
+```
+./bin/yarn-session.sh -n 10 -tm 8192 -s 32
+```
+
+
+
+系统会使用 `conf/flink-conf.yaml`文件中的配置. 需要更改配置可以参考 [configuration guide](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html) 。
+
+YARN上的Flink将覆盖以下配置参数 `jobmanager.rpc.address` (因为JobManager可能会分配到不同的机器上运行), `taskmanager.tmp.dirs` (我们使用YARN给出的tmp目录) 以及 `parallelism.default` 如果已经指定插槽数
+
+I如果不希望通过更改配置文件来设置配置参数，则可以选择通过`-D`标志传递动态属性。例如通过如下方式传递参数: `-Dfs.overwrite-files=true -Dtaskmanager.network.memory.min=536346624`.
+
+例子中启动了11个容器（即使只请求了10个容器）因为ApplicationMaster和Job Manager还有一个额外的容器。
+
+在YARN群集中部署Flink后，YARN会显示JobManager的详细信息。
+
+按下CTRL + C 或者输入 ‘stop’可以停止YARN会话。
+
+如果群集上资源从租，Flink会按照请求来启动容器。大多数YARN调度账户考虑所请求容器的内存量，还有些账户会思考vcores的数量。 默认情况下，vcores的数量等于处理slots (`-s`) 参数。 调整 [`yarn.containers.vcores`](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html#yarn-containers-vcores) 可以自定义vcores的数值。 为了使此参数起作用，需要在群集中启用CPU调度。
+
+#### 分离YARN 会话
+
+如果不想让Flink程序一直运行，可以启动一个 _分离_ YARN 会话。参数称为 `-d` or `--detached`。
+
+在这种情况下，Flink YARN客户端将仅向群集提交Flink，然后自行关闭。需要注意的是，在这种情况下，无法使用Flink停止YARN会话，需要使用 YARN 工具 (`yarn application -kill &lt;appId&gt;`) 来停止YARN会话。
+
+####  附加到已有Session
+
+使用以下命令启动会话
+
+
+
+```
+./bin/yarn-session.sh
+```
+
+
+
+命令将会显示一下内容
+
+
+
+```
+Usage:
+   Required
+     -id,--applicationId <yarnAppId> YARN application Id
+```
+
+
+
+正如前面所述, 必须设置`YARN_CONF_DIR` 和 `HADOOP_CONF_DIR` 环境变量来确保可以读取YARN和HDFS配置。
+
+**举例:** 输入以下命令以附加到正在运行的Flink YARN会话 `application_1463870264508_0029`:
+
+
+
+```
+./bin/yarn-session.sh -id application_1463870264508_0029
+```
+
+
+
+附加到正在运行的会话使用YARN ResourceManager来指定JobManagerRPC端口。
+
+按下CTRL + C 或者输入 ‘stop’可以停止YARN会话。
+
+### 向Flink提交作业
+
+使用以下命令将Flink程序提交到YARN群集:
+
+
+
+```
+./bin/flink
+```
+
+
+
+参数可以参阅 [command-line client](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/cli.html)文档。
+
+该命令将会按照如下显示：
+
+
+
+```
+[...]
+Action "run" compiles and runs a program.
+
+  Syntax: run [OPTIONS] <jar-file> <arguments>
+  "run" action arguments:
+     -c,--class <classname>           Class with the program entry point ("main"
+                                      method or "getPlan()" method. Only needed
+                                      if the JAR file does not specify the class
+                                      in its manifest.
+     -m,--jobmanager <host:port>      Address of the JobManager (master) to
+                                      which to connect. Use this flag to connect
+                                      to a different JobManager than the one
+                                      specified in the configuration.
+     -p,--parallelism <parallelism>   The parallelism with which to run the
+                                      program. Optional flag to override the
+                                      default value specified in the
+                                      configuration
+```
+
+
+
+使用 _run_ 命令把作业提交给YARN。客户端能够确定JobManager的地址。 少数情况下可以通过 `-m` 参数来指定JobManager的地址。JobManager可以在YARN的控制台上可见
+
+**示例**
+
+
+
+```
+wget -O LICENSE-2.0.txt http://www.apache.org/licenses/LICENSE-2.0.txt
+hadoop fs -copyFromLocal LICENSE-2.0.txt hdfs:/// ...
+./bin/flink run ./examples/batch/WordCount.jar \
+        hdfs:///..../LICENSE-2.0.txt hdfs:///.../wordcount-result.txt
+```
+
+
+
+如果出现以下错误，请检查所有TaskManagers都已启动:
+
+
+
+```
+Exception in thread "main" org.apache.flink.compiler.CompilerException:
+    Available instances could not be determined from job manager: Connection timed out.
+```
+
+
+
+可以在JobManager Web界面中检查TaskManagers的数量。接口的地址打印在YARN会话控制台中。
+
+如果一分钟后TaskManagers没有出现，则需要使用日志文件确认问题。
+
+## 在 YARN上运行单作业
+
+上面的文档描述了如何在YARN环境中启动Flink集群。也可以通过YARN中启动Flink来执行单个作业。
+
+需要注意客户端可以通过 `-yn` 参数来指定 TaskManagers的数量
+
+**_示例:_**
+
+
+
+```
+./bin/flink run -m yarn-cluster -yn 2 ./examples/batch/WordCount.jar
+```
+
+
+
+YARN会话的命令行选项也可用于 `./bin/flink`.以 `y` 或者 `yarn` 为前缀
+
+注意: 可以为每个作业使用不同的配置目录`FLINK_CONF_DIR`，需要这样配置时，请拷贝 `conf`目录，然后分发到不同节点上，并修改对应的配置
+
+注意：当对一个分离的YARN会话(`-yd`)使用`-m yarn-cluster`参数时，应用程序将不会从ExecutionEnvironment.execute（）调用获得任何累加器结果或异常！
+
+### jars 和 Classpath
+
+默认情况下，Flink将在运行单个作业时将用户jar包含到系统类路径中。可以使用 `yarn.per-job-cluster.include-user-jar` 参数控制.
+
+当设置为 `DISABLED` 时，Flink会使用用户类路径中的jar包
+
+可以通过以下之一来控制类路径中的user-jar位置:
+
+*   `ORDER`: (默认) 根据字典顺序将jar添加到系统类路径。
+*   `FIRST`: 将jar添加到系统类路径的开头。
+*   `LAST`: 将jar添加到系统类路径的末尾。
+
+## Flink 在 YARN上的故障恢复
+
+运行在YARN上的Flink可以通过配置参数来控制发生故障时的行为，相关配置可以在 `conf/flink-conf.yaml` 文件配置或使用 `-D` 参数指定。
+
+*   `yarn.reallocate-failed`: Flink是否应重新分配失败的TaskManager容器。默认值：true。
+*   `yarn.maximum-failed-containers`: ApplicationMaster在YARN会话失败之前接受的最大失败容器数，默认值：最初请求的TaskManagers (`-n`)的数量。
+*   `yarn.application-attempts`: ApplicationMaster (+ TaskManager ) 尝试次数，如果此值设置为1（默认值），则当Application master失败时，整个YARN会话将失败。较高的值指定YARN重新启动ApplicationMaster的次数。
+
+## 调试失败的YARN会话
+
+Flink YARN会话部署失败的原因有很多。Hadoop配置错误（HDFS权限，YARN配置），版本不兼容（在Cloudera Hadoop上运行Flink与vanilla Hadoop依赖关系）或其他错误。
+
+### 日志文件
+
+在部署期间失败，调试时比较有用的一个手段就是YARN日志聚合，参阅 [YARN 日志聚合](http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/). 在` yarn-site.xml` 中配置 `yarn.log-aggregation-enable` 为true可以启用配置。 启用后，用户可以使用以下命令检索（失败的）YARN会话的所有日志文件。
+
+```
+yarn logs -applicationId <application ID>
+```
+
+
+
+需要注意的是，会话结束后需要几秒钟才会显示日志。
+
+### YARN Client 终端 和 Web 界面
+
+如果在运行期间发生错误，Flink YARN客户端还会在终端中打印错误消息（例如，如果TaskManager在一段时间后停止工作）。
+
+除此之外，还有YARN Resource Manager Web界面（默认情况下在端口8088上）可以看到详情。端口由`yarn.resourcemanager.webapp.address`配置指定。
+
+界面可以看到日志文件以运行YARN应用程序，显示失败应用程序和一些错误信息。
+
+## 构建特定版本的YARN客户端
+
+使用Hortonworks，Cloudera或MapR等公司的Hadoop发行版的用户可能必须针对其特定版本的Hadoop（HDFS）和YARN构建Flink。相关内容可以查阅 [构建指南](//ci.apache.org/projects/flink/flink-docs-release-1.7/flinkDev/building.html) 。
+
+## 在防火墙后面的YARN上运行Flink
+
+某些YARN群集使用防火墙来控制群集与网络其余部分之间的网络流量。在这些设置中，Flink作业只能从群集网络内（防火墙后面）提交到YARN会话。这种情况下需要放开Flink应用的相关端口。
+
+目前两个与提交作业相关的服务:
+
+*    JobManager (ApplicationMaster in YARN)
+*   在JobManager中运行的BlobServer。
+
+向Flink提交作业时，BlobServer会将带有用户代码的jar分发给所有工作节点（TaskManagers）。JobManager自己接收作业并触发执行。
+
+用于指定端口的两个配置参数如下：
+
+*   `yarn.application-master.port`
+*   `blob.server.port`
+
+这两个配置选项可以配置，单个端口（例如：“50010”），范围（“50000-50025”）或两者的组合（“50010,50011,50020-50025,50050-50075”）。
+
+(Hadoop使用类似的机制，比如配置`yarn.app.mapreduce.am.job.client.port-range`.)
+
+## Flink和YARN的内部交互机制
+
+本节简要介绍Flink和YARN如何交互。
+
+![](img/FlinkOnYarn.svg)
+
+ARN客户端需要访问Hadoop配置用来连接到YARN资源管理器和HDFS。使用以下策略确定Hadoop配置：
+
+*   按顺序测试 `YARN_CONF_DIR`, `HADOOP_CONF_DIR` 或者 `HADOOP_CONF_PATH` 是否存在，如果其中一个变量存在，则使用配置中的路径来读取配置。
+*   如果上面的都不存在 (YARN设置正确不应该这样)，则客户端正在使用 `HADOOP_HOME` 环境变量。 如果设置了则尝试访问 `$HADOOP_HOME/etc/hadoop` (Hadoop 2) 和 `$HADOOP_HOME/conf` (Hadoop 1)路径来读取配置。
+
+启动新的Flink YARN会话时，客户端首先检查所请求的资源（容器和内存）是否可用。之后，将包含Flink和配置的jar上传到HDFS（步骤1）。
+
+客户端的下一步是请求（步骤2）YARN容器以启动 _ApplicationMaster_ (步骤 3)。由于客户端将配置和jar文件注册为容器的资源，因此在该特定机器上运行的YARN的NodeManager将负责准备容器（例如，下载文件）。完成后，将启动 _ApplicationMaster_ (AM) 。
+
+ _JobManager_ 和 AM 在同一容器内，运行一旦它们成功启动，AM就知道JobManager（它自己的主机）的地址。它正在为TaskManagers生成一个新的Flink配置文件（以便它们可以连接到JobManager）。该文件也自动上传到HDFS中， _AM_ 还提供了Flink的WEB界面。YARN代码分配的所有端口都是 _临时端口_，这样用户可以并行执行多个Flink YARN会话。
+
+之后，AM开始为Flink的TaskManagers分配容器，这将从HDFS下载jar文件和修改后的配置。完成这些步骤后，即可建立Flink并准备接受作业。
+

doc/flink-17-doc-zh/107.md

+
+
+# Mesos 设置
+
+## 背景
+
+Mesos实现包含两个组件：Application Master和Worker。Worker是简单的TaskManagers，由应用程序设置的环境进行参数化。Mesos实现中最复杂的组件是应用程序主机。应用程序主机当前托管以下组件：
+
+### Mesos 调度程序
+
+调度程序负责向Mesos注册，请求资源和启动工作节点。调度程序不断向Mesos报告以确保框架处于正常状态。为了验证群集的运行状况，调度程序监视生成的worker并将其标记为失败，并在必要时重新启动它们。
+
+Flink的Mesos调度程序本身目前不具备高可用性。但是，它会在Zookeeper中保存有关其状态（例如配置，工作者列表）的所有必要信息。在出现故障时，它依赖于外部系统来启动新的调度程序。然后，调度程序将再次向Mesos注册并完成协调阶段。在协调阶段，调度程序接收正在运行的工作程序节点的列表。它将这些与Zookeeper中恢复的信息进行匹配，并确保在故障之前将群集恢复到状态。
+
+### 工作服务器
+
+工件服务器负责向工作节点提供资源。资源可以是从Flink二进制文件或者是配置文件。例如，在非容器化环境中，工件服务器将提供Flink二进制文件和需要依赖的文件。
+
+### Flink’s 的调度程序和web页面
+
+Flink提供了Web界面为监视，为作业提交和监控交互提供了便利(参阅 [FLIP-6](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077)).
+
+### 启动脚本和配置覆盖
+
+启动脚本提供了一种配置和启动应用程序主机的方法。然后，工作节点继承所有的配置。这是使用配置覆盖来实现的。配置覆盖提供了一种从环境变量和配置文件推断配置的方法，这些配置文件将发送到工作节点。
+
+## DC/OS
+
+本文涉及到 [DC/OS](https://dcos.io) 它是具有复杂应用程序管理层的Mesos分发。它预装了Marathon，这是一种监控应用程序并在发生故障时保持其状态的服务。
+
+如果没有正在运行的DC/OS集群，可以参考 [如何安装DC/OS](https://dcos.io/install/).
+
+安装DC / OS群集后，可以通过DC / OS Universe安装Flink。在搜索提示中，只需搜索Flink。或者，可以使用DC / OS CLI：
+
+```
+dcos package install flink 
+```
+
+详细信息请参阅 [DC/OS 示例](https://github.com/dcos/examples/tree/master/1.8/flink).
+
+## 无 DC/OS的Mesos
+
+也没在没有DC/OS的环境中运行 Mesos。
+
+### 安装 Mesos
+
+参阅 [MEsos安装指南](http://mesos.apache.org/getting-started/)来完成安装。
+
+安装完成后，需要创建 `MESOS_HOME/etc/mesos/masters` 和 `MESOS_HOME/etc/mesos/slaves`文件来配置主机诶单和从节点。文件中每行包含一个主机名(要求可以通过ssh访问到这些节点)。
+
+接下来需要创建 `MESOS_HOME/etc/mesos/mesos-master-env.sh` 文件，或者在同目录下找到模板，修改模板内容。以下内容是必须的
+
+```
+export MESOS_work_dir=WORK_DIRECTORY 
+```
+
+并建议取消注释
+
+```
+export MESOS_log_dir=LOGGING_DIRECTORY 
+```
+
+配置Mesos代理，需要创建 `MESOS_HOME/etc/mesos/mesos-agent-env.sh` 文件，或在同目录下修改模板。
+
+```
+export MESOS_master=MASTER_HOSTNAME:MASTER_PORT 
+```
+
+并取消注释
+
+```
+export MESOS_log_dir=LOGGING_DIRECTORY
+export MESOS_work_dir=WORK_DIRECTORY 
+```
+
+#### Mesos 类库
+
+In order to run Java applications with Mesos you have to export `MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so` on Linux. Under Mac OS X you have to export `MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.dylib`.
+
+#### Deploying Mesos
+
+要使用Mesos运行Java应用程序， 请使用脚本 `MESOS_HOME/sbin/mesos-start-cluster.sh`. 停止Mesost则执行 `MESOS_HOME/sbin/mesos-stop-cluster.sh`. 更多关于部署脚本可以参阅[部署脚本](http://mesos.apache.org/documentation/latest/deploy-scripts/).
+
+### 安装 Marathon
+
+或者可以安装Marathon参阅 [Marathon安装](https://mesosphere.github.io/marathon/docs/) 即可开启高可用，参阅 [高可用模式](#high-availability).
+
+### 预安装Flink 与 Docker/Mesos 容器
+
+可以在所有Mesos主节点和代理节点上安装Flink。或者在部署期间从Flink网站拉取二进制文件，并在启动应用程序主服务器之前应用自定义配置。使用Docker容器来管理Flink二进制文件和配置会是个更方便，更易于维护的方法。
+
+通过以下配置控制的：
+
+```
+mesos.resourcemanager.tasks.container.type: mesos _or_ docker 
+```
+
+如果设置为 `docker`，需要指定镜像名称
+
+```
+mesos.resourcemanager.tasks.container.image.name: image_name 
+```
+
+### Standalone
+
+Flink的 `/bin` 目录下可以找到两个脚本用来管理Mesos集群中的Flink进程
+
+1.  `mesos-appmaster.sh` 启动Mesos master，负责注册Mesos调度程序，还负责启动工作节点。
+
+2.  `mesos-taskmanager.sh` Mesos工作进程的入口点。不需要执行此脚本。它由Mesos工作节点自动启动以启动新的TaskManager。
+
+执行 `mesos-appmaster.sh` 脚本需要在`flink-conf.yaml`中定义 `mesos.master`或者通过 `-Dmesos.master=…` 传递给Java进程
+
+执行`mesos-appmaster.sh`时，它将在执行脚本的机器上创建一个JobManager。TaskManager将作为Mesos集群中的Mesos任务运行。
+
+#### 基本配置
+
+可以通过传递给Mesos应用程序主机的Java属性完全参数化Mesos应用程序。还允许指定常规Flink配置参数。例如
+
+```
+bin/mesos-appmaster.sh \
+    -Dmesos.master=master.foobar.org:5050 \
+    -Djobmanager.heap.mb=1024 \
+    -Djobmanager.rpc.port=6123 \
+    -Drest.port=8081 \
+    -Dmesos.resourcemanager.tasks.mem=4096 \
+    -Dtaskmanager.heap.mb=3500 \
+    -Dtaskmanager.numberOfTaskSlots=2 \
+    -Dparallelism.default=10 
+```
+
+**Note:** 如果Flink处于 [legacy mode](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html#legacy), 还需要定义TaskManager数量 参阅[`mesos.initial-tasks`](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html#mesos-initial-tasks).
+
+### 高可用
+
+还需要运行Marathon或Apache Aurora之类的服务，来确保节点或进程出现故障时重新启动Flink主进程。此外还需要一些额外的Zookeeper配置，参阅 [Flink 文档-高可用](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/jobmanager_high_availability.html).
+
+#### Marathon
+
+执行 `bin/mesos-appmaster.sh` 脚本来启动Marathon，文件内还包含了一些Flink集群的配置参数。
+
+以下是Marathon的示例配置：
+
+```
+{
+    "id": "flink",
+    "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 -Djobmanager.rpc.port=6123 -Drest.port=8081 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1",
+    "cpus": 1.0,
+    "mem": 1024
+} 
+```
+
+使用Marathon运行Flink时，包含JobManager的整个Flink集群将作为Mesos集群中的Mesos任务运行。
+
+### 配置参数
+
+有关Mesos特定配置的列表，请参阅 [Mesos配置](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html#mesos)。
+

doc/flink-17-doc-zh/108.md

+
+
+# Docker安装
+
+[Docker](https://www.docker.com) 是一个流行的运行时容器。 Docker Hub上有用于Apache Flink的Docker镜像，可用于部署Flink群集。Flink源还有用于部署作业集群的镜像。
+
+## Flink集群
+
+Flink群集可用于运行多个作业。部署后，多个作业可逐个提交到集群。
+
+### Docker 镜像
+
+ [Flink Docker repository](https://hub.docker.com/_/flink/) 是Flink托管在 Docker Hub上的地址，提供了Flink1.2.1版本以上的镜像。
+
+镜像支持Hadoop和scala的环境组合，为了方便通tag别名来区分。
+
+从Flink 1.5版本以后, 镜像tags省略了Hadoop版本 (例如 `-hadoop28`) 对应于不包含Hadoop发行版的Flink的无Hadoop版本。
+
+举例，下面的别名可以使用: _(`1.5.y` 表示Flink 1.5的最新版本)_
+
+*   `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
+*   `flink:1.5` → `flink:1.5.y-scala_2.11`
+*   `flink:1.5-hadoop27` → `flink:1.5.y-hadoop27-scala_2.11`
+
+**注意:**  Docker镜像是由个人方式提供的社区项目。不是Apache Flink PMC的正式版本。
+
+## Flink作业集群
+
+Flink作业集群是运行单个作业的专用集群。这项是镜像内容的一部分，不需要额外的操作。
+
+### Docker 镜像
+
+Flink作业集群镜像需要包含用户代码jar。因此，需要为每个作业单独构建镜像。 `flink-container` 模块包含了一个 `build.sh` 脚本，用于创建此类镜像，详情可参阅 [instructions](https://github.com/apache/flink/blob/release-1.7/flink-container/docker/README.md) 。
+
+## Flink 和 Docker Compose
+
+[Docker Compose](https://docs.docker.com/compose/) 是一种在本地运行一组Docker容器的便捷方式。
+
+Github上提供了 [安装](https://github.com/docker-flink/examples/blob/master/docker-compose.yml) 和 [作业集群](https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-compose.yml) 的配置文件示例
+
+### 用法
+
+*   启动集群
+
+    ```
+     docker-compose up 
+    ```
+
+*   后台启动集群
+
+    ```
+     docker-compose up -d 
+    ```
+
+*   调整集群中为 _N_个 TaskManagers
+
+    ```
+     docker-compose scale taskmanager=<N> 
+    ```
+
+*   关闭集群
+
+    ```
+     docker-compose kill 
+    ```
+
+Flink集群运行时，可以通过 [http://localhost:8081](http://localhost:8081)来访问WEB UI，并通过UI来提交作业。
+
+通过命令行将作业提交到Flink群集，必须将JAR复制到JobManager容器中并从那里提交作业。
+
+例如:
+
+```
+$ JOBMANAGER_CONTAINER=$(docker ps --filter name=jobmanager --format={{.ID}})
+$ docker cp path/to/jar "$JOBMANAGER_CONTAINER":/job.jar
+$ docker exec -t -i "$JOBMANAGER_CONTAINER" flink run /job.jar 
+```
+

doc/flink-17-doc-zh/109.md

+
+
+# Kubernetes 设置
+
+本章介绍如何在 [Kubernetes](https://kubernetes.io)上部署Flink作业
+
+## 安装 Kubernetes
+
+按照 [Kubernetes’ setup guide](https://kubernetes.io/docs/setup/) 来部署kubernetes集群，如果需要在本地运行Kubernetes，建议使用 [MiniKube](https://kubernetes.io/docs/setup/minikube/).
+
+**注意:** 如果使用MiniKube，请确保在部署前执行 `minikube ssh 'sudo ip link set docker0 promisc on'` 命令，窦泽，Flink组件无法通过Kubernetes服务来引用自己
+
+## Kubernetes上的Flink集群
+
+Flink会话群集作为长期运行的Kubernetes部署而执行， 所以可以在集群中运行多个Flink作业，集群部署后，需要把任务逐个提交到集群。
+
+在Kubernetes上部署的Flink会话通常有三个组件:
+
+*   部署/作业运行JobManager
+*   部署TaskManagers池
+*   提供 JobManager’的 REST的 UI服务和端口
+
+### 在Kubernetes上部署Flink会话集群
+
+使用 `kubectl` 命令来定义 [session cluster](#session-cluster-resource-definitions)的资源 :
+
+```
+kubectl create -f jobmanager-service.yaml
+kubectl create -f jobmanager-deployment.yaml
+kubectl create -f taskmanager-deployment.yaml 
+```
+
+可以通过 `kubectl proxy` 来访问Flink UI:
+
+1.  在终端执行`kubectl proxy` 命令
+2.  在浏览器中输入 [http://localhost:8001/api/v1/namespaces/default/services/flink-jobmanager:ui/proxy](http://localhost:8001/api/v1/namespaces/default/services/flink-jobmanager:ui/proxy) 即可访问
+
+需要停止Flink集群，使用 `kubectl`:
+
+```
+kubectl delete -f jobmanager-deployment.yaml
+kubectl delete -f taskmanager-deployment.yaml
+kubectl delete -f jobmanager-service.yaml 
+```
+
+## Kubernetes上的Flink集群
+
+Flink作业集群是运行单个作业的专用集群。是镜像的一部分，所以不需要额外的工作
+
+### 创建专用作业的镜像
+
+Flink作业集群映像需要包含启动集群的作业的用户代码jar，因此，需要为每个作业构建专用的镜像。按照 [说明](https://github.com/apache/flink/blob/release-1.7/flink-container/docker/README.md) 来构建Docker镜像
+
+### 在Kubernetes上部署Flink作业集群
+
+在Kubernetes上部署作业集群，请参阅 [instructions](https://github.com/apache/flink/blob/release-1.7/flink-container/kubernetes/README.md#deploy-flink-job-cluster).
+
+## 集群的高级部署
+
+GitHub 上提供了早期版本的[Flink Helm chart](https://github.com/docker-flink/examples) 。
+
+## 附录
+
+### 会话集群的资源文件
+
+在 [on Docker Hub](https://hub.docker.com/r/_/flink/)上可以通过 `flink:latest`标签来找到已经构建好的镜像，镜像是从 [Github repository](https://github.com/docker-flink/docker-flink)这里构建的。
+
+`jobmanager-deployment.yaml`
+
+
+
+```
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  name: flink-jobmanager
+spec:
+  replicas: 1
+  template:
+    metadata:
+      labels:
+        app: flink
+        component: jobmanager
+    spec:
+      containers:
+      - name: jobmanager
+        image: flink:latest
+        args:
+        - jobmanager
+        ports:
+        - containerPort: 6123
+          name: rpc
+        - containerPort: 6124
+          name: blob
+        - containerPort: 6125
+          name: query
+        - containerPort: 8081
+          name: ui
+        env:
+        - name: JOB_MANAGER_RPC_ADDRESS
+          value: flink-jobmanager
+```
+
+
+
+`taskmanager-deployment.yaml`
+
+
+
+```
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  name: flink-taskmanager
+spec:
+  replicas: 2
+  template:
+    metadata:
+      labels:
+        app: flink
+        component: taskmanager
+    spec:
+      containers:
+      - name: taskmanager
+        image: flink:latest
+        args:
+        - taskmanager
+        ports:
+        - containerPort: 6121
+          name: data
+        - containerPort: 6122
+          name: rpc
+        - containerPort: 6125
+          name: query
+        env:
+        - name: JOB_MANAGER_RPC_ADDRESS
+          value: flink-jobmanager
+```
+
+
+
+`jobmanager-service.yaml`
+
+
+
+```
+apiVersion: v1
+kind: Service
+metadata:
+  name: flink-jobmanager
+spec:
+  ports:
+  - name: rpc
+    port: 6123
+  - name: blob
+    port: 6124
+  - name: query
+    port: 6125
+  - name: ui
+    port: 8081
+  selector:
+    app: flink
+    component: jobmanager
+```
+
+
+

doc/flink-17-doc-zh/11.md

+
+
+# 配置依赖项、连接器、库
+
+每个Flink应用程序都依赖于一组Flink库。至少，应用程序依赖于Flink API。许多应用程序还依赖于某些连接器库(如Kafka、Cassandra等)。在运行Flink应用程序时(无论是在分布式部署中还是在用于测试的IDE中)，Flink运行时库也必须可用。
+## Flink核心与应用依赖关系
+与大多数运行用户定义应用程序的系统一样，Flink中有两大类依赖关系和库：
+
+*   **Flink核心依赖关系**: Flink本身包括一组运行系统所需的类和依赖关系，例如协调、联网、检查点、故障转移、API、操作（如窗口）、资源管理等。所有这些类和依赖项的集合构成Flink运行时的核心，并且在启动Flink应用程序时必须存在。
+
+    这些核心类和依赖项被打包在 `flink-dist` jar中。它们是Flinks `lib` 文件夹的一部分，也是Flink容器基本图像的一部分。认为这些依赖关系类似于Java的核心库（`rt.jar`、`charset.jar`等）。），其中包含`String`和`List`等类。
+
+    Flink核心依赖项不包含任何连接器或库（CEP、SQL、ML等）为了避免默认情况下类路径中有过多的依赖项和类。事实上，我们试图保持核心依赖项尽可能小，以保持默认类路径的小，并避免依赖冲突。
+
+*   The **用户应用程序依赖项** 是特定用户应用程序需要的所有连接器、格式或库。
+
+    用户应用程序通常打包到 _application jar_ 中，其中包含应用程序代码以及所需的连接器和库依赖项。
+
+    用户应用程序依赖关系显式不包括FlinkDataSet/DataStreamAPI和运行时依赖关系，因为它们已经是Flink的CoreDependencies的一部分。
+## 设置项目：基本依赖项
+
+每个Flink应用程序都需要作为API依赖关系的最小值来进行开发。对于Maven，您可以使用[JavaProject Template](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/projectsetup/java_api_quickstart.html)或[ScalaProject Template](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/projectsetup/scala_api_quickstart.html)]来创建具有这些初始依赖项的程序框架。
+
+手动设置项目时，您需要为Java/ScalaAPI添加以下依赖项(这里是Maven语法，但同样的依赖项也适用于其他构建工具(Gradle、SBT等)。也是。
+
+<figure class="highlight">
+
+```
+<dependency>
+  <groupId>org.apache.flink</groupId>
+  <artifactId>flink-java</artifactId>
+  <version>1.7.1</version>
+  <scope>provided</scope>
+</dependency>
+<dependency>
+  <groupId>org.apache.flink</groupId>
+  <artifactId>flink-streaming-java_2.11</artifactId>
+  <version>1.7.1</version>
+  <scope>provided</scope>
+</dependency>
+```
+
+</figure>
+
+<figure class="highlight">
+
+```
+<dependency>
+  <groupId>org.apache.flink</groupId>
+  <artifactId>flink-scala_2.11</artifactId>
+  <version>1.7.1</version>
+  <scope>provided</scope>
+</dependency>
+<dependency>
+  <groupId>org.apache.flink</groupId>
+  <artifactId>flink-streaming-scala_2.11</artifactId>
+  <version>1.7.1</version>
+  <scope>provided</scope>
+</dependency>
+```
+
+</figure>
+
+**重要:** 请注意，所有这些依赖项的作用域都设置为_provided_。这意味着需要它们来编译，但是不应该将它们打包到项目的应用程序jar文件中-这些依赖项是FlinkCore依赖项，这些依赖项在任何设置中都已经可用。
+
+强烈建议将依赖项保持在作用域 _provided_ 中。如果它们不设置为 _provided_ ，最好的情况是结果JAR变得过大，因为它还包含所有Flink核心依赖项。最坏的情况是，添加到应用程序的jar文件中的Flink核心依赖项与您自己的一些依赖版本发生冲突（通常通过倒类加载来避免）。
+
+**关于IntelliJ:** 要使应用程序在IntelliJIDEA中运行，Flink依赖关系需要在 SCOPE_COMPILE__ 而不是 _REVITY_ 中声明。否则，IntelliJ将不会将它们添加到类路径中，使用“NoClassDefFountError”执行in-IDE将失败。为了避免将依赖关系范围声明为_COMPILE_(不建议这样做，见上文)，上面链接的Java和Scala项目模板使用了一个技巧：它们添加了一个配置文件，当应用程序在IntelliJ中运行时有选择地激活它，然后将依赖关系提升到Scope_COMPILE_，而不影响JAR文件的打包。
+
+## 添加连接器和库依赖关系
+
+大多数应用程序需要特定的连接器或库来运行，例如卡夫卡、卡桑德拉等的连接器。这些连接器不是Flink的核心依赖项的一部分，因此必须作为依赖项添加到应用程序中。
+
+下面是一个示例，将Kafka 0.10的连接器添加为依赖项(Maven语法)：
+
+<figure class="highlight">
+
+```
+<dependency>
+    <groupId>org.apache.flink</groupId>
+    <artifactId>flink-connector-kafka-0.10_2.11</artifactId>
+    <version>1.7.1</version>
+</dependency>
+```
+
+</figure>
+
+我们建议将应用程序代码及其所有必需的依赖项打包到一个 _jar-with-dependencies_，我们称之为 _application jar_。应用程序JAR可以提交给已经在运行的Flink集群，也可以添加到Flink应用程序容器映像中。
+
+从[JavaProject Template](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/projectsetup/java_api_quickstart.html)或[scala项目Template](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/projectsetup/scala_api_quickstart.html)]创建的项目被配置为在运行 `mvn clean package` 时自动将应用程序依赖项包含到应用程序JAR中。对于没有从这些模板中设置的项目，我们建议添加Maven Shade插件(如下面的附录所示)，以构建具有所有所需依赖项的应用程序JAR。
+
+**重要:** 要使Maven(和其他构建工具)正确地将依赖项打包到应用程序JAR中，这些应用程序依赖项必须在SCOPE_COMPILE_(与核心依赖项不同，必须在Scope_Providing_)中指定)。
+
+## Scala 版本
+
+Scala版本(2.10、2.11、2.12等)不是二进制兼容的。因此，Scala2.11的Flink不能与使用Scala2.12的应用程序一起使用。
+
+依赖于Scala的所有Flink依赖关系都是Suffix的，Scala版本为它们构建，例如`flink-streaming-scala_2.11`。
+只有Java的开发人员可以选择任何Scala版本，Scala开发人员需要选择与其应用程序Scala版本匹配的Scala版本。
+有关如何为特定Scala版本构建Flink的详细信息，请参阅[Build guide](//ci.apache.org/projects/flink/flink-docs-release-1.7/flinkDev/building.html#scala-versions)]。
+**Note:** 由于Scala2.12中的重大中断更改，Flink 1.5目前只为Scala2.11构建。我们的目标是在下一个版本中增加对Scala2.12的支持。
+
+## Hadoop Dependencies
+
+**一般规则：永远不必将Hadoop依赖项直接添加到应用程序中。** _(唯一的例外是在Flink的Hadoop兼容性包装器中使用现有Hadoop输入/输出格式时)_
+
+如果您想在Hadoop中使用Flink，您需要一个包含Hadoop依赖项的Flink设置，而不是将Hadoop作为应用程序依赖项添加。有关详细信息，请参阅[Hadoop设置Guide](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/hadoop.html)]。
+这种设计有两个主要原因：
+*  一些Hadoop交互发生在Flink的核心中，可能是在用户应用程序启动之前，例如为检查点设置HDFS、通过Hadoop的Kerberos令牌进行身份验证或在纱线上部署。
+*   Flink的反向类加载方法从核心依赖项中隐藏了许多传递依赖项。这不仅适用于Flink自己的核心依赖项，而且也适用于Hadoop在安装过程中的依赖关系。这样，应用程序可以使用相同依赖项的不同版本，而不会遇到依赖冲突(相信我们，这很重要，因为Hadoops依赖树很大)。
+如果在IDE内部测试或开发过程中需要Hadoop依赖项(例如，对于HDFS访问)，请将这些依赖项配置为与依赖关系范围类似的 _test_ 或 _Providing_ 。
+## 附录：用依赖关系构建JAR的模板
+要构建包含声明的连接器和库所需的所有依赖项的应用程序JAR，可以使用以下阴影插件定义：
+<figure class="highlight">
+
+```
+<build>
+	<plugins>
+		<plugin>
+			<groupId>org.apache.maven.plugins</groupId>
+			<artifactId>maven-shade-plugin</artifactId>
+			<version>3.0.0</version>
+			<executions>
+				<execution>
+					<phase>package</phase>
+					<goals>
+						<goal>shade</goal>
+					</goals>
+					<configuration>
+						<artifactSet>
+							<excludes>
+								<exclude>com.google.code.findbugs:jsr305</exclude>
+								<exclude>org.slf4j:*</exclude>
+								<exclude>log4j:*</exclude>
+							</excludes>
+						</artifactSet>
+						<filters>
+							<filter>
+								<!-- Do not copy the signatures in the META-INF folder.
+								Otherwise, this might cause SecurityExceptions when using the JAR. -->
+								<artifact>*:*</artifact>
+								<excludes>
+									<exclude>META-INF/*.SF</exclude>
+									<exclude>META-INF/*.DSA</exclude>
+									<exclude>META-INF/*.RSA</exclude>
+								</excludes>
+							</filter>
+						</filters>
+						<transformers>
+							<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
+								<mainClass>my.programs.main.clazz</mainClass>
+							</transformer>
+						</transformers>
+					</configuration>
+				</execution>
+			</executions>
+		</plugin>
+	</plugins>
+</build>
+```
+
+</figure>
+

doc/flink-17-doc-zh/110.md

+Amazon 服务 (AWS)
+
+Amazon Web Services 提供可以运行Flink的云计算服务。
+
+## EMR: Elastic MapReduce
+
+[Amazon Elastic MapReduce](https://aws.amazon.com/elasticmapreduce/) (Amazon EMR) 是一种Web服务，可以轻松快速地设置Hadoop集群。因为它负责设置所有内容，所以在AWS上运行Flink **非常推荐**，使用这种方式 。
+
+### 标准 EMR 安装
+
+Flink是Amazon EMR上受支持的应用程序。 [亚马逊的文档](http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-flink.html) 描述了如何配置Flink、创建和监控集群以及处理作业。
+
+### 自定义 EMR 安装
+
+Amazon EMR 会定期更新到最新版本， 也可以在EMR集群中安装不同版本的Flink
+
+**创建 EMR 集群**
+
+EMR文档包含了 [创建 EMR集群](http://docs.aws.amazon.com/ElasticMapReduce/latest/ManagementGuide/emr-gs-launch-sample-cluster.html)。通过该文档可以安装任何版本的EMR，不需要安装EMR中的_All Applications_部分，但是_Core Hadoop_是必须的。
+
+注意 访问S3存储需要配置IAM角色，文档请参阅 [配置 IAM角色](http://docs.aws.amazon.com/ElasticMapReduce/latest/ManagementGuide/emr-iam-roles.html) 。
+
+**在EMR集群中安装Flink**
+
+在创建集群之后，连接到主节点并安装Flink，文档请参阅 [连接到主节点](http://docs.aws.amazon.com/ElasticMapReduce/latest/ManagementGuide/emr-connect-master-node.html) 。
+
+1.  转到 [下载页](http://flink.apache.org/downloads.html) 并 **下载与EMR集群中Hadoop版本匹配的二进制版本** ， 例如， Hadoop 2.7 for EMR releases 4.3.0, 4.4.0, or 4.5.0.
+2.  解压Flink，**设置Hadoop配置目录**后，可以运行Flink作业，文档请参阅 [Flink jobs via YARN](yarn_setup.html) :
+
+
+
+```
+HADOOP_CONF_DIR=/etc/hadoop/conf ./bin/flink run -m yarn-cluster -yn 1 examples/streaming/WordCount.jar
+```
+
+
+
+## S3: 简单存储
+
+[Amazon Simple Storage Service](http://aws.amazon.com/s3/) (Amazon S3)为各种实例提供云对象存储，S3可以作为Flink的数据**输入** 或 **输出** ，或者是为 [streaming **state backends**](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/state/state_backends.html) 、以及YARN提供对象存储。
+
+通过以下格式指定路径来使用常规文件等S3对象：
+
+
+
+```
+s3://<your-bucket>/<endpoint>
+```
+
+
+
+endpoint 是单个文件或目录，例如：
+
+
+
+```
+// 从S3中读取
+env.readTextFile("s3://<bucket>/<endpoint>");
+
+// 写入到S3
+stream.writeAsText("s3://<bucket>/<endpoint>");
+
+// 使用S3作为 FsStatebackend
+env.setStateBackend(new FsStateBackend("s3://<your-bucket>/<endpoint>"));
+```
+
+
+
+上文的示例并没有列出全部情况，也可以在其他地方使用S3存储，包括 [高可用安装](../jobmanager_high_availability.html) 或 [RocksDBStateBackend](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/state/state_backends.html#the-rocksdbstatebackend)等 任何Flink期望的文件地址。
+
+多数情况下我们使用  `flink-s3-fs-hadoop` 和 `flink-s3-fs-presto` 作为S3在Hadoop 上的实现 ，例如，使用S3作为YARN的资源存储目录，可能需要设置特定的Hadoop S3 FileSystem实现。两种方式如下所述。
+
+### Shaded Hadoop/Presto S3 文件系统 (推荐)
+
+**Note:** 如果在EMR上运行Flink，则无需手动配置，文档可以参阅  [Flink on EMR](#emr-elastic-mapreduce).
+
+要使用 `flink-s3-fs-hadoop` 或 `flink-s3-fs-presto`，co在启动Flink之前将相应的JAR文件从 `opt` 目录拷贝到Flink的 `lib` 目录下，例如
+
+
+
+```
+cp ./opt/flink-s3-fs-presto-1.7.1.jar ./lib/
+```
+
+
+
+ `flink-s3-fs-hadoop` 和 `flink-s3-fs-presto` 注册了不同的文件URI `s3://` 实现, `flink-s3-fs-hadoop` 还注册了 `s3a://`， `flink-s3-fs-presto` 注册了 `s3p://`，可以同时使用这些不同的URI。
+
+#### 配置访问凭据
+
+设置S3 文件系统实现之后，需要保证Flink程序可以访问S3存储。
+
+##### 身份和访问管理 (IAM) (推荐)
+
+比较推荐使用身份和访问管理 (IAM)来设置Access key，可以通过IAM提供的功能来让Flink程序安全的访问S3存储。  详细可以参阅[文档身份和访问管理 (IAM)](http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html)。具体用户角色权限控制可以参考 [IAM Roles](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
+
+如果正确设置此选项，则可以在AWS中管理对S3的访问，并且Flink访问S3不需要任何keys
+
+##### Access Keys (不推荐)
+
+还可以通过 **access key**来对S3进行访问。相关内容可以参阅 [ IAM的角色](https://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-safer-way-to-distribute-AWS-credentials-to-EC2).
+
+这种情况下，需要在Flink的 `flink-conf.yaml`文件中同时配置`s3.access-key` 和 `s3.secret-key` 
+
+
+
+```
+s3.access-key: your-access-key
+s3.secret-key: your-secret-key
+```
+
+
+
+### Hadoop提供的S3文件系统-手动设置
+
+**Note:** 在EMR上运行Flink不需要配置，请参阅 [Flink on EMR](#emr-elastic-mapreduce).
+
+因为这个设置比较复杂，所以除非是有其他需求，否则，还是建议才用上面的方式来实现。例如，修改Hadoop的 `core-site.xml`文件中的 `fs.defaultFS`配置来把S3作为存储目录.
+
+#### 设置 S3 文件系统
+
+ S3相关内容可以参阅 [Hadoop’s S3 FileSystem clients](https://wiki.apache.org/hadoop/AmazonS3):
+
+1.  `S3AFileSystem` (**推荐** Hadoop 2.7 及其以上版本使用): 用于在内部使用Amazon SDK读取和写入常规文件的文件系统。没有最大文件大小并且与IAM角色一起使用。
+2.  `NativeS3FileSystem` (Hadoop 2.6 及一起版本): 用于读写常规文件的文件系统。最大对象大小为5GB，不适用于IAM角色。
+
+##### `S3AFileSystem` (推荐)
+
+推荐使用的S3 FileSystem实现。它在内部使用Amazon的SDK并与IAM角色一起使用 (参阅 [配置访问凭据](#configure-access-credentials-1)).
+
+需要修改Flink中Hadoop的配置，配置文件 `core-site.xml`:
+
+
+
+```
+<configuration>
+
+<property>
+  <name>fs.s3.impl</name>
+  <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
+</property>
+
+<!-- Comma separated list of local directories used to buffer
+     large results prior to transmitting them to S3\. -->
+<property>
+  <name>fs.s3a.buffer.dir</name>
+  <value>/tmp</value>
+</property>
+
+</configuration>
+```
+
+
+
+这里注册 `S3AFileSystem` 作为 `s3a://` URI开头的文件实现.
+
+##### `NativeS3FileSystem`
+
+此文件系统仅限于最大5GB的文件，并且不适用于IAM角色(参阅 [配置访问凭据](#configure-access-credentials-1))，所以需要在配置文件中配置AWS的access key。
+
+需要修改Flink中Hadoop的配置，配置文件`core-site.xml`:
+
+
+
+```
+<property>
+  <name>fs.s3.impl</name>
+  <value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value>
+</property>
+```
+
+
+
+这里注册 `NativeS3FileSystem` 作为 `s3://` URI开头文件的实现.
+
+#### Hadoop 配置
+
+可以采用如下两种方式指定 [Hadoop 配置](../config.html#hdfs) 把Flink指向Hadoop的配置目录
+
+*   设置环境变量`HADOOP_CONF_DIR`, 
+*   设置  `flink-conf.yaml`文件中的 `fs.hdfs.hadoopconf`:
+
+
+
+```
+fs.hdfs.hadoopconf: /path/to/etc/hadoop
+```
+
+
+
+ `/path/to/etc/hadoop` 被注册为Hadoop的配置目录. Flink 会在目录下查找 `core-site.xml` 和 `hdfs-site.xml` 文件。
+
+#### 配置访问凭据
+
+**Note:** 如果在EMR上直接运行Flink，则无需额外配置，请参阅 [Flink on EMR](#emr-elastic-mapreduce).
+
+Flink中使用 S3 存储时，需要保证Flink可以访问到S3存储。
+
+##### 身份和访问管理 (IAM) (推荐)
+
+比较推荐使用身份和访问管理 (IAM)来设置Access key，可以通过IAM提供的功能来让Flink程序安全的访问S3存储。  详细可以参阅[文档身份和访问管理 (IAM)](http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html)。具体用户角色权限控制可以参考 [IAM 角色介绍](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
+
+如果正确设置此选项，则可以在AWS中管理对S3的访问，并且Flink访问S3不需要任何keys
+
+##### Access Keys (不推荐)
+
+还可以通过 **access key**来对S3进行访问。相关内容可以参阅 [ IAM的角色](https://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-safer-way-to-distribute-AWS-credentials-to-EC2).
+
+请注意，这只适用于 `S3AFileSystem` 而不是 `NativeS3FileSystem`.
+
+##### 通过Access Keys 访问 `S3AFileSystem` (不推荐)
+
+可以用过 **Access Keys**来授权对S3存储的访问。 但是这种操作并不推荐，请参阅 [IAM 角色介绍](https://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-safer-way-to-distribute-AWS-credentials-to-EC2).
+
+对于 `S3AFileSystem` 需要在Hadoop的`core-site.xml`文件中同时配置 `fs.s3a.access.key` 和 `fs.s3a.secret.key` :
+
+
+
+```
+<property>
+  <name>fs.s3a.access.key</name>
+  <value></value>
+</property>
+
+<property>
+  <name>fs.s3a.secret.key</name>
+  <value></value>
+</property>
+```
+
+
+
+##### 通过Access Keys访问`NativeS3FileSystem` (不推荐)
+
+可以用过 **Access Keys**来授权对S3存储的访问。 此文件系统已经过时，最好使用 `S3AFileSystem`来代替，详情参阅 [IAM 角色要求](https://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-safer-way-to-distribute-AWS-credentials-to-EC2).
+
+ 对于`NativeS3FileSystem` 需要在Hadoop的 `core-site.xml`文件中同事配置 `fs.s3.awsAccessKeyId` 和 `fs.s3.awsSecretAccessKey` :
+
+
+
+```
+<property>
+  <name>fs.s3.awsAccessKeyId</name>
+  <value></value>
+</property>
+
+<property>
+  <name>fs.s3.awsSecretAccessKey</name>
+  <value></value>
+</property>
+```
+
+
+
+#### 提供 S3 FileSystem 依赖
+
+**Note:** 在EMR上运行Flink则无需配置，请参阅 [Flink on EMR](#emr-elastic-mapreduce).
+
+Hadoop的S3 FileSystem客户端打包在 `hadoop-aws` jar中(Hadoop 2.6及其以后)。需要将JAR及其所有依赖项添加到Flink的类路径中，即Job和TaskManagers的类路径。根据使用的FileSystem实现以及使用的Flink和Hadoop版本，需要添加不同的依赖项（请参阅下文）。
+
+有多种方法可以将JAR添加到Flink的类路径中，最简单的方法就是将JAR放在Flink的 `lib` 目录下。 拷贝 `hadoop-aws` JAR 和他的所有依赖复制到lib下，还可以把一个目录指定为 `HADOOP_CLASSPATH` 环境变量。
+
+##### Flink for Hadoop 2.7
+
+根据您使用的操作系统，请添加以下依赖项。可以在`hadoop-2.7/share/hadoop/tools/lib`找到一部分:
+
+*   `S3AFileSystem`:
+    *   `hadoop-aws-2.7.3.jar`
+    *   `aws-java-sdk-s3-1.11.183.jar` 及其依赖:
+        *   `aws-java-sdk-core-1.11.183.jar`
+        *   `aws-java-sdk-kms-1.11.183.jar`
+        *   `jackson-annotations-2.6.7.jar`
+        *   `jackson-core-2.6.7.jar`
+        *   `jackson-databind-2.6.7.jar`
+        *   `joda-time-2.8.1.jar`
+        *   `httpcore-4.4.4.jar`
+        *   `httpclient-4.5.3.jar`
+*   `NativeS3FileSystem`:
+    *   `hadoop-aws-2.7.3.jar`
+    *   `guava-11.0.2.jar`
+
+注意 `hadoop-common` 是Flink的一部分, 但 Guava不是。
+
+##### Flink for Hadoop 2.6
+
+根据您使用的操作系统，请添加以下依赖项。可以在`hadoop-2.6/share/hadoop/tools/lib`找到一部分:
+
+*   `S3AFileSystem`:
+    *   `hadoop-aws-2.6.4.jar`
+    *   `aws-java-sdk-1.7.4.jar` and its dependencies:
+        *   `jackson-annotations-2.1.1.jar`
+        *   `jackson-core-2.1.1.jar`
+        *   `jackson-databind-2.1.1.jar`
+        *   `joda-time-2.2.jar`
+        *   `httpcore-4.2.5.jar`
+        *   `httpclient-4.2.5.jar`
+*   `NativeS3FileSystem`:
+    *   `hadoop-aws-2.6.4.jar`
+    *   `guava-11.0.2.jar`
+
+注意 `hadoop-common` 是Flink的一部分, 但 Guava不是。
+
+##### Flink for Hadoop 2.4及其以下版本
+
+2.4及其以下版本只支持 `NativeS3FileSystem`相应依赖已经包含在`hadoop-common`中了，所以不需要额外添加依赖。
+
+## 常见问题
+
+下面列出了在AWS上使用Flink时的部分常见问题。
+
+### Missing S3 FileSystem Configuration
+
+如果作业提交失败，并显示 `No file system found with scheme s3` 这说明S3文件系统的配置不正确，需要检查文件配置，参照  [Shaded Hadoop/Presto S3 文件系统 ](#shaded-hadooppresto-s3-file-systems-recommended) 或者 [Hadoop基本配置](#set-s3-filesystem) 保证配置正确性
+
+
+
+```
+org.apache.flink.client.program.ProgramInvocationException: The program execution failed:
+  Failed to submit job cd927567a81b62d7da4c18eaa91c3c39 (WordCount Example) [...]
+Caused by: org.apache.flink.runtime.JobException: Creating the input splits caused an error:
+  No file system found with scheme s3, referenced in file URI 's3://<bucket>/<endpoint>'. [...]
+Caused by: java.io.IOException: No file system found with scheme s3,
+  referenced in file URI 's3://<bucket>/<endpoint>'.
+    at o.a.f.core.fs.FileSystem.get(FileSystem.java:296)
+    at o.a.f.core.fs.Path.getFileSystem(Path.java:311)
+    at o.a.f.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:450)
+    at o.a.f.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:57)
+    at o.a.f.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:156)
+```
+
+
+
+### AWS Access Key ID and Secret Access Key Not Specified
+
+如果作业失败并显示异常 `AWS Access Key ID and Secret Access Key must be specified as the username or password`, 未正确设置您的访问凭据。有关如何配置请参阅 [shaded Hadoop/Presto](#configure-access-credentials) or [Hadoop基本配置](#configure-access-credentials-1) 。
+
+
+
+```
+org.apache.flink.client.program.ProgramInvocationException: The program execution failed:
+  Failed to submit job cd927567a81b62d7da4c18eaa91c3c39 (WordCount Example) [...]
+Caused by: java.io.IOException: The given file URI (s3://<bucket>/<endpoint>) points to the
+  HDFS NameNode at <bucket>, but the File System could not be initialized with that address:
+  AWS Access Key ID and Secret Access Key must be specified as the username or password
+  (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId
+  or fs.s3n.awsSecretAccessKey properties (respectively) [...]
+Caused by: java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must
+  be specified as the username or password (respectively) of a s3 URL, or by setting
+  the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties (respectively) [...]
+    at o.a.h.fs.s3.S3Credentials.initialize(S3Credentials.java:70)
+    at o.a.h.fs.s3native.Jets3tNativeFileSystemStore.initialize(Jets3tNativeFileSystemStore.java:80)
+    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:606)
+    at o.a.h.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
+    at o.a.h.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
+    at o.a.h.fs.s3native.$Proxy6.initialize(Unknown Source)
+    at o.a.h.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:330)
+    at o.a.f.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:321)
+```
+
+
+
+### ClassNotFoundException: NativeS3FileSystem/S3AFileSystem Not Found
+
+看到此异常，表示S3 FileSystem不是Flink的类路径的一部分请参阅 [S3 文件存储依赖](#provide-s3-filesystem-dependency) 。
+
+
+
+```
+Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3native.NativeS3FileSystem not found
+  at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2186)
+  at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getHadoopWrapperClassNameForFileSystem(HadoopFileSystem.java:460)
+  at org.apache.flink.core.fs.FileSystem.getHadoopWrapperClassNameForFileSystem(FileSystem.java:352)
+  at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:280)
+  at org.apache.flink.core.fs.Path.getFileSystem(Path.java:311)
+  at org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:450)
+  at org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:57)
+  at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:156)
+  ... 25 more
+Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3native.NativeS3FileSystem not found
+  at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2154)
+  at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2178)
+  ... 32 more
+Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3native.NativeS3FileSystem not found
+  at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2060)
+  at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2152)
+  ... 33 more
+```
+
+
+
+### IOException: `400: Bad Request`
+
+如果您已正确配置所有内容，但是请求却显示 `Bad Request` 异常 **and** S3 bucket在 `eu-central-1`可用区， 可能是因为S3客户端不支持，请参阅[Amazon’s signature version 4](http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html).
+
+
+
+```
+[...]
+Caused by: java.io.IOException: s3://<bucket-in-eu-central-1>/<endpoint> : 400 : Bad Request [...]
+Caused by: org.jets3t.service.impl.rest.HttpException [...]
+```
+
+
+
+或
+
+
+
+```
+com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS Service: Amazon S3, AWS Request ID: [...], AWS Error Code: null, AWS Error Message: Bad Request, S3 Extended Request ID: [...]
+```
+
+
+
+Hadoop/Presto S3 存储不会有问题，但是 Hadoop-provided S3 file systems会有。Hadoop versions 大于 2.7.2 运行于 `NativeS3FileSystem` (依赖 `JetS3t 0.9.0` 版本 [&gt;= 0.9.4](http://www.jets3t.org/RELEASE_NOTES.html)) 都会受影响，部分 `S3AFileSystem`也可能出现该异常。
+
+除了更改可用区之外,可以参阅亚马逊的 [requesting signature version 4 for request authentication](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version)进行修改, 例如. 在 `flink-conf.yaml` 中添加JVM参数(参阅 [配置](../config.html#common-options)):
+
+
+
+```
+env.java.opts: -Dcom.amazonaws.services.s3.enableV4
+```
+
+
+
+### NullPointerException at org.apache.hadoop.fs.LocalDirAllocator
+
+此异常通常是由跳过本地缓冲区目录配置引起   `S3AFileSystem`的`fs.s3a.buffer.dir`配置。参阅 [S3A文件系统配置](#s3afilesystem-recommended) 来正确配置.
+
+
+
+```
+[...]
+Caused by: java.lang.NullPointerException at
+o.a.h.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268) at
+o.a.h.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344) at
+o.a.h.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:416) at
+o.a.h.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator.java:198) at
+o.a.h.fs.s3a.S3AOutputStream.<init>(S3AOutputStream.java:87) at
+o.a.h.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:410) at
+o.a.h.fs.FileSystem.create(FileSystem.java:907) at
+o.a.h.fs.FileSystem.create(FileSystem.java:888) at
+o.a.h.fs.FileSystem.create(FileSystem.java:785) at
+o.a.f.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:404) at
+o.a.f.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:48) at
+... 25 more
+```
+
+
+

doc/flink-17-doc-zh/111.md

+
+
+# 谷歌计算引擎设置
+
+# Google Compute Engine Setup
+
+本文提供如何在[谷歌计算引擎](https://cloud.google.com/compute/)集群上全面自动的基于Hadoop 1 或者Hadoop 2设置Flink，谷歌的 [bdutil](https://cloud.google.com/hadoop/bdutil) 使启动和部署基于Hadoop的flink集群成为可能。请按照以下步骤操作。
+
+This documentation provides instructions on how to setup Flink fully automatically with Hadoop 1 or Hadoop 2 on top of a [Google Compute Engine](https://cloud.google.com/compute/) cluster. This is made possible by Google’s [bdutil](https://cloud.google.com/hadoop/bdutil) which starts a cluster and deploys Flink with Hadoop. To get started, just follow the steps below.
+
+# 先决条件
+
+# Prerequisites
+
+## 安装谷歌云套件
+
+## Install Google Cloud SDK
+
+请按照指示如何安装[谷歌云套件](https://cloud.google.com/sdk/)。特别是，请务必使用以下命令通过谷歌云进行身份验证：
+```
+gcloud auth login 
+```
+
+Please follow the instructions on how to setup the [Google Cloud SDK](https://cloud.google.com/sdk/). In particular, make sure to authenticate with Google Cloud using the following command:
+
+```
+gcloud auth login 
+```
+
+## 安装bdutil
+
+## Install bdutil
+
+目前还没有包含Flink扩展的bdutil发布。 但是，您可以从[GitHub](https://github.com/GoogleCloudPlatform/bdutil)获得Flink支持的最新版本的bdutil：
+
+```
+git clone https://github.com/GoogleCloudPlatform/bdutil.git 
+```
+下载源代码后，切换到新创建的`bdutil`目录并继续执行后续步骤。
+
+At the moment, there is no bdutil release yet which includes the Flink extension. However, you can get the latest version of bdutil with Flink support from [GitHub](https://github.com/GoogleCloudPlatform/bdutil):
+
+```
+git clone https://github.com/GoogleCloudPlatform/bdutil.git 
+```
+
+After you have downloaded the source, change into the newly created `bdutil` directory and continue with the next steps.
+
+# 在Google Compute Engine上部署Flink
+
+# Deploying Flink on Google Compute Engine
+
+## 设置一个桶
+
+## Set up a bucket
+
+如果尚未执行此操作，请为bdutil配置和暂存文件创建存储桶。 可以使用gsutil创建一个新存储桶：
+
+```
+gsutil mb gs://<bucket_name> 
+```
+
+If you have not done so, create a bucket for the bdutil config and staging files. A new bucket can be created with gsutil:
+
+```
+gsutil mb gs://<bucket_name> 
+```
+
+## 调整bdutil配置
+
+## Adapt the bdutil config
+
+要使用bdutil部署Flink，请至少调整bdutil_env.sh中的以下变量：
+
+```
+CONFIGBUCKET="<bucket_name>"
+PROJECT="<compute_engine_project_name>"
+NUM_WORKERS=<number_of_workers>
+
+# set this to 'n1-standard-2' if you're using the free trial
+GCE_MACHINE_TYPE="<gce_machine_type>"
+
+# for example: "europe-west1-d"
+GCE_ZONE="<gce_zone>" 
+```
+
+To deploy Flink with bdutil, adapt at least the following variables in bdutil_env.sh.
+
+```
+CONFIGBUCKET="<bucket_name>"
+PROJECT="<compute_engine_project_name>"
+NUM_WORKERS=<number_of_workers>
+
+# set this to 'n1-standard-2' if you're using the free trial
+GCE_MACHINE_TYPE="<gce_machine_type>"
+
+# for example: "europe-west1-d"
+GCE_ZONE="<gce_zone>" 
+```
+
+## 调整Flink配置
+
+## Adapt the Flink config
+
+bdutil的Flink扩展为您处理配置。 您还可以在`extensions / flink / flink_env.sh`中调整配置变量。 如果您想进一步配置，请查看[配置Flink]（../ config.html）。 使用`bin / stop-cluster`和`bin / start-cluster`更改配置后，必须重新启动Flink。
+
+bdutil’s Flink extension handles the configuration for you. You may additionally adjust configuration variables in `extensions/flink/flink_env.sh`. If you want to make further configuration, please take a look at [configuring Flink](../config.html). You will have to restart Flink after changing its configuration using `bin/stop-cluster` and `bin/start-cluster`.
+
+## 使用Flink创建一个集群
+
+## Bring up a cluster with Flink
+
+要在Google Compute Engine上调出Flink群集，请执行：
+
+```
+./bdutil -e extensions/flink/flink_env.sh deploy 
+```
+
+To bring up the Flink cluster on Google Compute Engine, execute:
+
+```
+./bdutil -e extensions/flink/flink_env.sh deploy 
+```
+
+## 运行一个Flink示例作业：
+
+```
+./bdutil shell
+cd /home/hadoop/flink-install/bin
+./flink run ../examples/batch/WordCount.jar gs://dataflow-samples/shakespeare/othello.txt gs://<bucket_name>/output 
+```
+
+## Run a Flink example job:
+
+```
+./bdutil shell
+cd /home/hadoop/flink-install/bin
+./flink run ../examples/batch/WordCount.jar gs://dataflow-samples/shakespeare/othello.txt gs://<bucket_name>/output 
+```
+
+## 关闭你的集群
+
+## Shut down your cluster
+
+关闭集群就像执行下面一样简单
+
+```
+./bdutil -e extensions/flink/flink_env.sh delete 
+```
+
+Shutting down a cluster is as simple as executing
+
+```
+./bdutil -e extensions/flink/flink_env.sh delete 
+```
+

doc/flink-17-doc-zh/112.md

+
+
+# MapR Setup
+
+This documentation provides instructions on how to prepare Flink for YARN executions on a [MapR](https://mapr.com/) cluster.
+
+## Running Flink on YARN with MapR
+
+The instructions below assume MapR version 5.2.0\. They will guide you to be able to start submitting [Flink on YARN](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/yarn_setup.html) jobs or sessions to a MapR cluster.
+
+### Building Flink for MapR
+
+In order to run Flink on MapR, Flink needs to be built with MapR’s own Hadoop and Zookeeper distribution. Simply build Flink using Maven with the following command from the project root directory:
+
+
+
+```
+mvn clean install -DskipTests -Pvendor-repos,mapr \
+    -Dhadoop.version=2.7.0-mapr-1607 \
+    -Dzookeeper.version=3.4.5-mapr-1604
+```
+
+
+
+The `vendor-repos` build profile adds MapR’s repository to the build so that MapR’s Hadoop / Zookeeper dependencies can be fetched. The `mapr` build profile additionally resolves some dependency clashes between MapR and Flink, as well as ensuring that the native MapR libraries on the cluster nodes are used. Both profiles must be activated.
+
+By default the `mapr` profile builds with Hadoop / Zookeeper dependencies for MapR version 5.2.0, so you don’t need to explicitly override the `hadoop.version` and `zookeeper.version` properties. For different MapR versions, simply override these properties to appropriate values. The corresponding Hadoop / Zookeeper distributions for each MapR version can be found on MapR documentations such as [here](http://maprdocs.mapr.com/home/DevelopmentGuide/MavenArtifacts.html).
+
+### Job Submission Client Setup
+
+The client submitting Flink jobs to MapR also needs to be prepared with the below setups.
+
+Ensure that MapR’s JAAS config file is picked up to avoid login failures:
+
+
+
+```
+export JVM_ARGS=-Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf
+```
+
+
+
+Make sure that the `yarn.nodemanager.resource.cpu-vcores` property is set in `yarn-site.xml`:
+
+
+
+```
+<!-- in /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/yarn-site.xml -->
+
+<configuration>
+...
+
+<property>
+    <name>yarn.nodemanager.resource.cpu-vcores</name>
+    <value>...</value>
+</property>
+
+...
+</configuration>
+```
+
+
+
+Also remember to set the `YARN_CONF_DIR` or `HADOOP_CONF_DIR` environment variables to the path where `yarn-site.xml` is located:
+
+
+
+```
+export YARN_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/
+export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/
+```
+
+
+
+Make sure that the MapR native libraries are picked up in the classpath:
+
+
+
+```
+export FLINK_CLASSPATH=/opt/mapr/lib/*
+```
+
+
+
+If you’ll be starting Flink on YARN sessions with `yarn-session.sh`, the below is also required:
+
+
+
+```
+export CC_CLASSPATH=/opt/mapr/lib/*
+```
+
+
+
+## Running Flink with a Secured MapR Cluster
+
+_Note: In Flink 1.2.0, Flink’s Kerberos authentication for YARN execution has a bug that forbids it to work with MapR Security. Please upgrade to later Flink versions in order to use Flink with a secured MapR cluster. For more details, please see [FLINK-5949](https://issues.apache.org/jira/browse/FLINK-5949)._
+
+Flink’s [Kerberos authentication](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/security-kerberos.html) is independent of [MapR’s Security authentication](http://maprdocs.mapr.com/home/SecurityGuide/Configuring-MapR-Security.html). With the above build procedures and environment variable setups, Flink does not require any additional configuration to work with MapR Security.
+
+Users simply need to login by using MapR’s `maprlogin` authentication utility. Users that haven’t acquired MapR login credentials would not be able to submit Flink jobs, erroring with:
+
+
+
+```
+java.lang.Exception: unable to establish the security context
+Caused by: o.a.f.r.security.modules.SecurityModule$SecurityInstallException: Unable to set the Hadoop login user
+Caused by: java.io.IOException: failure to login: Unable to obtain MapR credentials
+```
+
+
+

doc/flink-17-doc-zh/113.md

+
+
+# Hadoop Integration
+
+## Configuring Flink with Hadoop Classpaths
+
+Flink will use the environment variable `HADOOP_CLASSPATH` to augment the classpath that is used when starting Flink components such as the Client, JobManager, or TaskManager. Most Hadoop distributions and cloud environments will not set this variable by default so if the Hadoop classpath should be picked up by Flink the environment variable must be exported on all machines that are running Flink components.
+
+When running on YARN, this is usually not a problem because the components running inside YARN will be started with the Hadoop classpaths, but it can happen that the Hadoop dependencies must be in the classpath when submitting a job to YARN. For this, it’s usually enough to do a
+
+
+
+```
+export HADOOP_CLASSPATH=`hadoop classpath`
+```
+
+
+
+in the shell. Note that `hadoop` is the hadoop binary and that `classpath` is an argument that will make it print the configured Hadoop classpath.
+

doc/flink-17-doc-zh/115.md

+
+
+# 检查点
+
+## 概览
+
+## Overview
+
+检查点通过恢复任务状态和流的位置使flink达到容错的特性，从而为flink任务提供与无故障执行相同的语句。
+
+Checkpoints make state in Flink fault tolerant by allowing state and the corresponding stream positions to be recovered, thereby giving the application the same semantics as a failure-free execution.
+
+查看[检查点](https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/state/checkpointing.html)如何在你的flink任务开启和设置checkpoints机制。
+
+See [Checkpointing](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/state/checkpointing.html) for how to enable and configure checkpoints for your program.
+
+## 检查点保留机制
+
+## Retained Checkpoints
+
+默认情况下，检查点不会保留，只有在任务失败的情况下才会使用。当程序被取消，它们也会被删除。但是你可以配置要保留的检查点。根据配置，当任务失败或取消时，检查点不会自动被清除，这样你可以通过检查点来恢复失败的任务。
+
+Checkpoints are by default not retained and are only used to resume a job from failures. They are deleted when a program is cancelled. You can, however, configure periodic checkpoints to be retained. Depending on the configuration these _retained_ checkpoints are _not_ automatically cleaned up when the job fails or is canceled. This way, you will have a checkpoint around to resume from if your job fails.
+
+
+
+```
+CheckpointConfig config = env.getCheckpointConfig();
+config.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
+```
+
+`ExternalizedCheckpointCleanup`： 模式指定了当你取消任务的时候会发生什么。
+
+*   **`ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION`**：当任务被取消保留检查点。注意在这种情况下，任务被取消后你必须手动去清除检查点状态。
+
+*   **`ExternalizedCheckpointCleanup.DELETE_ON_CANCELLATION`**: 当任务被取消删除检查点。只有当任务失败检查点的状态才会有效。 
+
+The `ExternalizedCheckpointCleanup` mode configures what happens with checkpoints when you cancel the job:
+
+*   **`ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION`**: Retain the checkpoint when the job is cancelled. Note that you have to manually clean up the checkpoint state after cancellation in this case.
+
+*   **`ExternalizedCheckpointCleanup.DELETE_ON_CANCELLATION`**: Delete the checkpoint when the job is cancelled. The checkpoint state will only be available if the job fails.
+
+### 目录结构
+
+### Directory Structure
+
+与[保存点](savepoints.html)相同，检查点文件包含一个元数据文件以及一些其他数据文件，具体取决于状态后端。元数据文件和数据文件保存的目录由`state.checkpoints.dir` 参数配置，也可以再每个任务的代码中单独指定。
+
+Similarly to [savepoints](savepoints.html), a checkpoint consists of a meta data file and, depending on the state backend, some additional data files. The meta data file and data files are stored in the directory that is configured via `state.checkpoints.dir` in the configuration files, and also can be specified for per job in the code.
+
+#### 通过配置文件设置全局路径 
+
+```
+state.checkpoints.dir: hdfs:///checkpoints/
+```
+
+#### Configure globally via configuration files
+
+
+
+```
+state.checkpoints.dir: hdfs:///checkpoints/
+```
+
+#### 在构造状态后端时为每一个任务配置检查点路径
+
+```
+env.setStateBackend(new RocksDBStateBackend("hdfs:///checkpoints-data/");
+```
+
+#### Configure for per job when constructing the state backend
+
+
+
+```
+env.setStateBackend(new RocksDBStateBackend("hdfs:///checkpoints-data/");
+```
+
+
+### 与保存点的差异
+
+### Difference to Savepoints
+
+检查点与保存点有一些不同，主要在
+
+*   使用状态后端特定的数据格式，可能是增量的。
+
+*   不支持flink的特定功能，例如重新缩放。
+
+Checkpoints have a few differences from [savepoints](savepoints.html). They
+
+*   use a state backend specific (low-level) data format, may be incremental.
+*   do not support Flink specific features like rescaling.
+
+### 通过保留的检查点恢复任务
+
+### Resuming from a retained checkpoint
+
+通过使用检查点的元数据文件，可以从检查点恢复作业，就像从保存点恢复一样（请参阅[保存点恢复指南](../cli.html#restore-a-savepoint)）。注意如果元数据文件不是自包含的，作业管理器需要访问它指向的数据文件（查看上面的[目录结构](#目录结构)）。
+
+```
+$ bin/flink run -s :checkpointMetaDataPath [:runArgs]
+```
+
+A job may be resumed from a checkpoint just as from a savepoint by using the checkpoint’s meta data file instead (see the [savepoint restore guide](../cli.html#restore-a-savepoint)). Note that if the meta data file is not self-contained, the jobmanager needs to have access to the data files it refers to (see [Directory Structure](#directory-structure) above).
+
+
+
+```
+$ bin/flink run -s :checkpointMetaDataPath [:runArgs]
+```
+
+
+

doc/flink-17-doc-zh/116.md

+
+# 保存点
+# Savepoints
+
+## 什么是保存点？保存点与检查点有什么不同？
+
+## What is a Savepoint? How is a Savepoint different from a Checkpoint?
+
+保存点是通过flink的[检查机制](//ci.apache.org/projects/flink/flink-docs-release-1.7/internals/stream_checkpointing.html)创建流作业执行状态的一致图像。你可以使用保存点停止、恢复、复制或者更新你的flink任务。保存点包含两部分：一个在稳定存储器（例如HDFS S3等）上的（通常很大）二进制文件目录以及（相对较小的）元数据文件。在稳定存储器上的文件表示执行状态图像的网络数据。保存点的元数据以绝对路径的形式包含指向作为savepoint一部分的稳定存储器上的所有文件的指针。
+
+A Savepoint is a consistent image of the execution state of a streaming job, created via Flink’s [checkpointing mechanism](//ci.apache.org/projects/flink/flink-docs-release-1.7/internals/stream_checkpointing.html). You can use Savepoints to stop-and-resume, fork, or update your Flink jobs. Savepoints consist of two parts: a directory with (typically large) binary files on stable storage (e.g. HDFS, S3, …) and a (relatively small) meta data file. The files on stable storage represent the net data of the job’s execution state image. The meta data file of a Savepoint contains (primarily) pointers to all files on stable storage that are part of the Savepoint, in form of absolute paths.
+
+**注意：** 为了允许程序和flink版本之间的升级，请务必检查以下[你的操作分配ID](#assigning-operator-ids)相关的部分。
+
+**Attention:** In order to allow upgrades between programs and Flink versions, it is important to check out the following section about [assigning IDs to your operators](#assigning-operator-ids).
+
+从概念上讲，flink的保存点和检查点的不同点在于从传统数据库系统中恢复日志的备份。检查点的主要目的是对于未知的任务失败提供了一个恢复机制。一个检查点的生命周期是由flink管理的。例如一个检查点的创建，授权和发布都是由flink指定的，而不是由用户。作为一种恢复和定期触发的方式，检查实现的两个主要设计目标是：1. 轻量级创建 2.尽可能快的恢复。针对这些目标的优化可以利用某些属性，例如，作业代码在执行尝试的时候不会改变。 检查点通常会在用户终止任务的时候被删除。（除非明确配置为保留的检查点）
+
+Conceptually, Flink’s Savepoints are different from Checkpoints in a similar way that backups are different from recovery logs in traditional database systems. The primary purpose of Checkpoints is to provide a recovery mechanism in case of unexpected job failures. A Checkpoint’s lifecycle is managed by Flink, i.e. a Checkpoint is created, owned, and released by Flink - without user interaction. As a method of recovery and being periodically triggered, two main design goals for the Checkpoint implementation are i) being as lightweight to create and ii) being as fast to restore from as possible. Optimizations towards those goals can exploit certain properties, e.g. that the job code doesn’t change between the execution attempts. Checkpoints are usually dropped after the job was terminated by the user (except if explicitly configured as retained Checkpoints).
+
+In contrast to all this, Savepoints are created, owned, and deleted by the user. Their use-case is for planned, manual backup and resume. For example, this could be an update of your Flink version, changing your job graph, changing parallelism, forking a second job like for a red/blue deployment, and so on. Of course, Savepoints must survive job termination. Conceptually, Savepoints can be a bit more expensive to produce and restore and focus more on portability and support for the previously mentioned changes to the job.
+
+Those conceptual differences aside, the current implementations of Checkpoints and Savepoints are basically using the same code and produce the same „format”. However, there is currently one exception from this, and we might introduce more differences in the future. The exception are incremental checkpoints with the RocksDB state backend. They are using some RocksDB internal format instead of Flink’s native savepoint format. This makes them the first instance of a more lightweight checkpointing mechanism, compared to Savepoints.
+
+## Assigning Operator IDs
+
+It is **highly recommended** that you adjust your programs as described in this section in order to be able to upgrade your programs in the future. The main required change is to manually specify operator IDs via the **`uid(String)`** method. These IDs are used to scope the state of each operator.
+
+
+
+```
+DataStream<String> stream = env.
+  // Stateful source (e.g. Kafka) with ID
+  .addSource(new StatefulSource())
+  .uid("source-id") // ID for the source operator
+  .shuffle()
+  // Stateful mapper with ID
+  .map(new StatefulMapper())
+  .uid("mapper-id") // ID for the mapper
+  // Stateless printing sink
+  .print(); // Auto-generated ID
+```
+
+
+
+If you don’t specify the IDs manually they will be generated automatically. You can automatically restore from the savepoint as long as these IDs do not change. The generated IDs depend on the structure of your program and are sensitive to program changes. Therefore, it is highly recommended to assign these IDs manually.
+
+### Savepoint State
+
+You can think of a savepoint as holding a map of `Operator ID -&gt; State` for each stateful operator:
+
+
+
+```
+Operator ID | State
+------------+------------------------
+source-id   | State of StatefulSource
+mapper-id   | State of StatefulMapper
+```
+
+
+
+In the above example, the print sink is stateless and hence not part of the savepoint state. By default, we try to map each entry of the savepoint back to the new program.
+
+## Operations
+
+You can use the [command line client](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/cli.html#savepoints) to _trigger savepoints_, _cancel a job with a savepoint_, _resume from savepoints_, and _dispose savepoints_.
+
+With Flink &gt;= 1.2.0 it is also possible to _resume from savepoints_ using the webui.
+
+### Triggering Savepoints
+
+When triggering a savepoint, a new savepoint directory is created where the data as well as the meta data will be stored. The location of this directory can be controlled by [configuring a default target directory](#configuration) or by specifying a custom target directory with the trigger commands (see the [`:targetDirectory` argument](#trigger-a-savepoint)).
+
+**Attention:** The target directory has to be a location accessible by both the JobManager(s) and TaskManager(s) e.g. a location on a distributed file-system.
+
+For example with a `FsStateBackend` or `RocksDBStateBackend`:
+
+
+
+```
+# Savepoint target directory
+/savepoints/
+
+# Savepoint directory
+/savepoints/savepoint-:shortjobid-:savepointid/
+
+# Savepoint file contains the checkpoint meta data
+/savepoints/savepoint-:shortjobid-:savepointid/_metadata
+
+# Savepoint state
+/savepoints/savepoint-:shortjobid-:savepointid/...
+```
+
+
+
+**Note:** Although it looks as if the savepoints may be moved, it is currently not possible due to absolute paths in the `_metadata` file. Please follow [FLINK-5778](https://issues.apache.org/jira/browse/FLINK-5778) for progress on lifting this restriction.
+
+Note that if you use the `MemoryStateBackend`, metadata _and_ savepoint state will be stored in the `_metadata` file. Since it is self-contained, you may move the file and restore from any location.
+
+**Attention:** It is discouraged to move or delete the last savepoint of a running job, because this might interfere with failure-recovery. Savepoints have side-effects on exactly-once sinks, therefore to ensure exactly-once semantics, if there is no checkpoint after the last savepoint, the savepoint will be used for recovery.
+
+#### Trigger a Savepoint
+
+
+
+```
+$ bin/flink savepoint :jobId [:targetDirectory]
+```
+
+
+
+This will trigger a savepoint for the job with ID `:jobId`, and returns the path of the created savepoint. You need this path to restore and dispose savepoints.
+
+#### Trigger a Savepoint with YARN
+
+
+
+```
+$ bin/flink savepoint :jobId [:targetDirectory] -yid :yarnAppId
+```
+
+
+
+This will trigger a savepoint for the job with ID `:jobId` and YARN application ID `:yarnAppId`, and returns the path of the created savepoint.
+
+#### Cancel Job with Savepoint
+
+
+
+```
+$ bin/flink cancel -s [:targetDirectory] :jobId
+```
+
+
+
+This will atomically trigger a savepoint for the job with ID `:jobid` and cancel the job. Furthermore, you can specify a target file system directory to store the savepoint in. The directory needs to be accessible by the JobManager(s) and TaskManager(s).
+
+### Resuming from Savepoints
+
+
+
+```
+$ bin/flink run -s :savepointPath [:runArgs]
+```
+
+
+
+This submits a job and specifies a savepoint to resume from. You may give a path to either the savepoint’s directory or the `_metadata` file.
+
+#### Allowing Non-Restored State
+
+By default the resume operation will try to map all state of the savepoint back to the program you are restoring with. If you dropped an operator, you can allow to skip state that cannot be mapped to the new program via `--allowNonRestoredState` (short: `-n`) option:
+
+
+
+```
+$ bin/flink run -s :savepointPath -n [:runArgs]
+```
+
+
+
+### Disposing Savepoints
+
+
+
+```
+$ bin/flink savepoint -d :savepointPath
+```
+
+
+
+This disposes the savepoint stored in `:savepointPath`.
+
+Note that it is possible to also manually delete a savepoint via regular file system operations without affecting other savepoints or checkpoints (recall that each savepoint is self-contained). Up to Flink 1.2, this was a more tedious task which was performed with the savepoint command above.
+
+### Configuration
+
+You can configure a default savepoint target directory via the `state.savepoints.dir` key. When triggering savepoints, this directory will be used to store the savepoint. You can overwrite the default by specifying a custom target directory with the trigger commands (see the [`:targetDirectory` argument](#trigger-a-savepoint)).
+
+
+
+```
+# Default savepoint target directory
+state.savepoints.dir: hdfs:///flink/savepoints
+```
+
+
+
+If you neither configure a default nor specify a custom target directory, triggering the savepoint will fail.
+
+**Attention:** The target directory has to be a location accessible by both the JobManager(s) and TaskManager(s) e.g. a location on a distributed file-system.
+
+## F.A.Q
+
+### Should I assign IDs to all operators in my job?
+
+As a rule of thumb, yes. Strictly speaking, it is sufficient to only assign IDs via the `uid` method to the stateful operators in your job. The savepoint only contains state for these operators and stateless operator are not part of the savepoint.
+
+In practice, it is recommended to assign it to all operators, because some of Flink’s built-in operators like the Window operator are also stateful and it is not obvious which built-in operators are actually stateful and which are not. If you are absolutely certain that an operator is stateless, you can skip the `uid` method.
+
+### What happens if I add a new operator that requires state to my job?
+
+When you add a new operator to your job it will be initialized without any state. Savepoints contain the state of each stateful operator. Stateless operators are simply not part of the savepoint. The new operator behaves similar to a stateless operator.
+
+### What happens if I delete an operator that has state from my job?
+
+By default, a savepoint restore will try to match all state back to the restored job. If you restore from a savepoint that contains state for an operator that has been deleted, this will therefore fail.
+
+You can allow non restored state by setting the `--allowNonRestoredState` (short: `-n`) with the run command:
+
+
+
+```
+$ bin/flink run -s :savepointPath -n [:runArgs]
+```
+
+
+
+### What happens if I reorder stateful operators in my job?
+
+If you assigned IDs to these operators, they will be restored as usual.
+
+If you did not assign IDs, the auto generated IDs of the stateful operators will most likely change after the reordering. This would result in you not being able to restore from a previous savepoint.
+
+### What happens if I add or delete or reorder operators that have no state in my job?
+
+If you assigned IDs to your stateful operators, the stateless operators will not influence the savepoint restore.
+
+If you did not assign IDs, the auto generated IDs of the stateful operators will most likely change after the reordering. This would result in you not being able to restore from a previous savepoint.
+
+### What happens when I change the parallelism of my program when restoring?
+
+If the savepoint was triggered with Flink &gt;= 1.2.0 and using no deprecated state API like `Checkpointed`, you can simply restore the program from a savepoint and specify a new parallelism.
+
+If you are resuming from a savepoint triggered with Flink &lt; 1.2.0 or using now deprecated APIs you first have to migrate your job and savepoint to Flink &gt;= 1.2.0 before being able to change the parallelism. See the [upgrading jobs and Flink versions guide](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/upgrading.html).
+
+### Can I move the Savepoint files on stable storage?
+
+The quick answer to this question is currently “no” because the meta data file references the files on stable storage as absolute paths for technical reasons. The longer answer is: if you MUST move the files for some reason there are two potential approaches as workaround. First, simpler but potentially more dangerous, you can use an editor to find the old path in the meta data file and replace them with the new path. Second, you can use the class SavepointV2Serializer as starting point to programmatically read, manipulate, and rewrite the meta data file with the new paths.
+

doc/flink-17-doc-zh/117.md

+
+# 状态后端
+
+# State Backends
+
+在[Data Stream API](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/datastream_api.html)写的程序一般会以不同的方式保存状态：
+
+*    Windows会在触发元素或聚合之前收集元素或聚合
+*    转换方法可能会使用键值对接口来存储值
+*    转换方法可能继承 `CheckpointedFunction` 接口来使其局部变量具有容错能力
+
+也可以参考API介绍中的[状态部分](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/state/index.html) 
+
+
+Programs written in the [Data Stream API](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/datastream_api.html) often hold state in various forms:
+
+*   Windows gather elements or aggregates until they are triggered
+*   Transformation functions may use the key/value state interface to store values
+*   Transformation functions may implement the `CheckpointedFunction` interface to make their local variables fault tolerant
+
+See also [state section](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/state/index.html) in the streaming API guide.
+
+当checkpointing被激活，这种状态被持久化在checkpoints中来防止数据丢死以及持续恢复。状态如何在内部表示，以及在检查点上如何及在何处持续取决于选择的 **后端状态**.
+
+When checkpointing is activated, such state is persisted upon checkpoints to guard against data loss and recover consistently. How the state is represented internally, and how and where it is persisted upon checkpoints depends on the chosen **State Backend**.
+
+## 后端状态可用
+
+## Available State Backends
+
+开箱即用，Flink捆绑了这些后端状态
+
+*   _MemoryStateBackend_
+*   _FsStateBackend_
+*   _RocksDBStateBackend_
+
+如果没有配置其他任何内容，系统将使用内存后端状态。
+
+Out of the box, Flink bundles these state backends:
+
+*   _MemoryStateBackend_
+*   _FsStateBackend_
+*   _RocksDBStateBackend_
+
+If nothing else is configured, the system will use the MemoryStateBackend.
+
+### 内存后端状态
+
+
+
+### The MemoryStateBackend
+
+The _MemoryStateBackend_ holds data internally as objects on the Java heap. Key/value state and window operators hold hash tables that store the values, triggers, etc.
+
+Upon checkpoints, this state backend will snapshot the state and send it as part of the checkpoint acknowledgement messages to the JobManager (master), which stores it on its heap as well.
+
+The MemoryStateBackend can be configured to use asynchronous snapshots. While we strongly encourage the use of asynchronous snapshots to avoid blocking pipelines, please note that this is currently enabled by default. To disable this feature, users can instantiate a `MemoryStateBackend` with the corresponding boolean flag in the constructor set to `false`(this should only used for debug), e.g.:
+
+
+
+```
+ new MemoryStateBackend(MAX_MEM_STATE_SIZE, false);
+```
+
+
+
+Limitations of the MemoryStateBackend:
+
+*   The size of each individual state is by default limited to 5 MB. This value can be increased in the constructor of the MemoryStateBackend.
+*   Irrespective of the configured maximal state size, the state cannot be larger than the akka frame size (see [Configuration](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html)).
+*   The aggregate state must fit into the JobManager memory.
+
+The MemoryStateBackend is encouraged for:
+
+*   Local development and debugging
+*   Jobs that do hold little state, such as jobs that consist only of record-at-a-time functions (Map, FlatMap, Filter, …). The Kafka Consumer requires very little state.
+
+### The FsStateBackend
+
+The _FsStateBackend_ is configured with a file system URL (type, address, path), such as “hdfs://namenode:40010/flink/checkpoints” or “file:///data/flink/checkpoints”.
+
+The FsStateBackend holds in-flight data in the TaskManager’s memory. Upon checkpointing, it writes state snapshots into files in the configured file system and directory. Minimal metadata is stored in the JobManager’s memory (or, in high-availability mode, in the metadata checkpoint).
+
+The FsStateBackend uses _asynchronous snapshots by default_ to avoid blocking the processing pipeline while writing state checkpoints. To disable this feature, users can instantiate a `FsStateBackend` with the corresponding boolean flag in the constructor set to `false`, e.g.:
+
+
+
+```
+ new FsStateBackend(path, false);
+```
+
+
+
+The FsStateBackend is encouraged for:
+
+*   Jobs with large state, long windows, large key/value states.
+*   All high-availability setups.
+
+### The RocksDBStateBackend
+
+The _RocksDBStateBackend_ is configured with a file system URL (type, address, path), such as “hdfs://namenode:40010/flink/checkpoints” or “file:///data/flink/checkpoints”.
+
+The RocksDBStateBackend holds in-flight data in a [RocksDB](http://rocksdb.org) database that is (per default) stored in the TaskManager data directories. Upon checkpointing, the whole RocksDB database will be checkpointed into the configured file system and directory. Minimal metadata is stored in the JobManager’s memory (or, in high-availability mode, in the metadata checkpoint).
+
+The RocksDBStateBackend always performs asynchronous snapshots.
+
+Limitations of the RocksDBStateBackend:
+
+*   As RocksDB’s JNI bridge API is based on byte[], the maximum supported size per key and per value is 2^31 bytes each. IMPORTANT: states that use merge operations in RocksDB (e.g. ListState) can silently accumulate value sizes > 2^31 bytes and will then fail on their next retrieval. This is currently a limitation of RocksDB JNI.
+
+The RocksDBStateBackend is encouraged for:
+
+*   Jobs with very large state, long windows, large key/value states.
+*   All high-availability setups.
+
+Note that the amount of state that you can keep is only limited by the amount of disk space available. This allows keeping very large state, compared to the FsStateBackend that keeps state in memory. This also means, however, that the maximum throughput that can be achieved will be lower with this state backend. All reads/writes from/to this backend have to go through de-/serialization to retrieve/store the state objects, which is also more expensive than always working with the on-heap representation as the heap-based backends are doing.
+
+RocksDBStateBackend is currently the only backend that offers incremental checkpoints (see [here](large_state_tuning.html)).
+
+Certain RocksDB native metrics are available but disabled by default, you can find full documentation [here](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html#rocksdb-native-metrics)
+
+## Configuring a State Backend
+
+The default state backend, if you specify nothing, is the jobmanager. If you wish to establish a different default for all jobs on your cluster, you can do so by defining a new default state backend in **flink-conf.yaml**. The default state backend can be overridden on a per-job basis, as shown below.
+
+### Setting the Per-job State Backend
+
+The per-job state backend is set on the `StreamExecutionEnvironment` of the job, as shown in the example below:
+
+
+
+```
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+env.setStateBackend(new FsStateBackend("hdfs://namenode:40010/flink/checkpoints"));
+```
+
+
+
+
+
+```
+val env = StreamExecutionEnvironment.getExecutionEnvironment()
+env.setStateBackend(new FsStateBackend("hdfs://namenode:40010/flink/checkpoints"))
+```
+
+
+
+### Setting Default State Backend
+
+A default state backend can be configured in the `flink-conf.yaml`, using the configuration key `state.backend`.
+
+Possible values for the config entry are _jobmanager_ (MemoryStateBackend), _filesystem_ (FsStateBackend), _rocksdb_ (RocksDBStateBackend), or the fully qualified class name of the class that implements the state backend factory [FsStateBackendFactory](https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsStateBackendFactory.java), such as `org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory` for RocksDBStateBackend.
+
+The `state.checkpoints.dir` option defines the directory to which all backends write checkpoint data and meta data files. You can find more details about the checkpoint directory structure [here](checkpoints.html#directory-structure).
+
+A sample section in the configuration file could look as follows:
+
+
+
+```
+# The backend that will be used to store operator state checkpoints
+
+state.backend: filesystem
+
+# Directory for storing checkpoints
+
+state.checkpoints.dir: hdfs://namenode:40010/flink/checkpoints
+```
+
+
+
+#### RocksDB State Backend Config Options
+
+| Key | Default | Description |
+| --- | --- | --- |
+| 
+
+##### state.backend.rocksdb.localdir
+
+ | (none) | The local directory (on the TaskManager) where RocksDB puts its files. |
+| 
+
+##### state.backend.rocksdb.timer-service.factory
+
+ | "HEAP" | This determines the factory for timer service state implementation. Options are either HEAP (heap-based, default) or ROCKSDB for an implementation based on RocksDB . |
+

doc/flink-17-doc-zh/120.md

+
+
+# Production Readiness Checklist
+
+## Production Readiness Checklist
+
+Purpose of this production readiness checklist is to provide a condensed overview of configuration options that are important and need **careful considerations** if you plan to bring your Flink job into **production**. For most of these options Flink provides out-of-the-box defaults to make usage and adoption of Flink easier. For many users and scenarios, those defaults are good starting points for development and completely sufficient for “one-shot” jobs.
+
+However, once you are planning to bring a Flink application to production the requirements typically increase. For example, you want your job to be (re-)scalable and to have a good upgrade story for your job and new Flink versions.
+
+In the following, we present a collection of configuration options that you should check before your job goes into production.
+
+### Set maximum parallelism for operators explicitly
+
+Maximum parallelism is a configuration parameter that is newly introduced in Flink 1.2 and has important implications for the (re-)scalability of your Flink job. This parameter, which can be set on a per-job and/or per-operator granularity, determines the maximum parallelism to which you can scale operators. It is important to understand that (as of now) there is **no way to change** this parameter after your job has been started, except for restarting your job completely from scratch (i.e. with a new state, and not from a previous checkpoint/savepoint). Even if Flink would provide some way to change maximum parallelism for existing savepoints in the future, you can already assume that for large states this is likely a long running operation that you want to avoid. At this point, you might wonder why not just to use a very high value as default for this parameter. The reason behind this is that high maximum parallelism can have some impact on your application’s performance and even state sizes, because Flink has to maintain certain metadata for its ability to rescale which can increase with the maximum parallelism. In general, you should choose a max parallelism that is high enough to fit your future needs in scalability, but keeping it as low as possible can give slightly better performance. In particular, a maximum parallelism higher that 128 will typically result in slightly bigger state snapshots from the keyed backends.
+
+Notice that maximum parallelism must fulfill the following conditions:
+
+`0 < parallelism <= max parallelism <= 2^15`
+
+You can set the maximum parallelism by `setMaxParallelism(int maxparallelism)`. By default, Flink will choose the maximum parallelism as a function of the parallelism when the job is first started:
+
+*   `128` : for all parallelism <= 128.
+*   `MIN(nextPowerOfTwo(parallelism + (parallelism / 2)), 2^15)` : for all parallelism > 128.
+
+### Set UUIDs for operators
+
+As mentioned in the documentation for [savepoints](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/state/savepoints.html), users should set uids for operators. Those operator uids are important for Flink’s mapping of operator states to operators which, in turn, is essential for savepoints. By default operator uids are generated by traversing the JobGraph and hashing certain operator properties. While this is comfortable from a user perspective, it is also very fragile, as changes to the JobGraph (e.g. exchanging an operator) will result in new UUIDs. To establish a stable mapping, we need stable operator uids provided by the user through `setUid(String uid)`.
+
+### Choice of state backend
+
+Currently, Flink has the limitation that it can only restore the state from a savepoint for the same state backend that took the savepoint. For example, this means that we can not take a savepoint with a memory state backend, then change the job to use a RocksDB state backend and restore. While we are planning to make backends interoperable in the near future, they are not yet. This means you should carefully consider which backend you use for your job before going to production.
+
+In general, we recommend using RocksDB because this is currently the only state backend that supports large states (i.e. state that exceeds the available main memory) and asynchronous snapshots. From our experience, asynchronous snapshots are very important for large states because they do not block the operators and Flink can write the snapshots without stopping stream processing. However, RocksDB can have worse performance than, for example, the memory-based state backends. If you are sure that your state will never exceed main memory and blocking the stream processing to write it is not an issue, you **could consider** to not use the RocksDB backends. However, at this point, we **strongly recommend** using RocksDB for production.
+
+### Config JobManager High Availability(HA)
+
+The JobManager coordinates every Flink deployment. It is responsible for both _scheduling_ and _resource management_.
+
+By default, there is a single JobManager instance per Flink cluster. This creates a _single point of failure_ (SPOF): if the JobManager crashes, no new programs can be submitted and running programs fail.
+
+With JobManager High Availability, you can recover from JobManager failures and thereby eliminate the _SPOF_. We **strongly recommend** you configure [high availability](//ci.apache.org/projects/flink/flink-docs-release-1.7/ops/jobmanager_high_availability.html) for production.
+

doc/flink-17-doc-zh/133.md

+
+
+# Debugging Windows & Event Time
+
+## Monitoring Current Event Time
+
+Flink’s [event time](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/event_time.html) and watermark support is a powerful feature for handling out-of-order events. However, it’s harder to understand what exactly is going on because the progress of time is tracked within the system.
+
+Low watermarks of each task can be accessed through Flink web interface or [metrics system](//ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html).
+
+Each Task in Flink exposes a metric called `currentLowWatermark` that represents the lowest watermark received by this task. This long value represents the “current event time”. The value is calculated by taking the minimum of all watermarks received by upstream operators. This means that the event time tracked with watermarks is always dominated by the furthest-behind source.
+
+The low watermark metric is accessible **using the web interface**, by choosing a task in the metric tab, and selecting the `<taskNr>.currentLowWatermark` metric. In the new box you’ll now be able to see the current low watermark of the task.
+
+Another way of getting the metric is using one of the **metric reporters**, as described in the documentation for the [metrics system](//ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html). For local setups, we recommend using the JMX metric reporter and a tool like [VisualVM](https://visualvm.github.io/).
+
+## Handling Event Time Stragglers
+
+*   Approach 1: Watermark stays late (indicated completeness), windows fire early
+*   Approach 2: Watermark heuristic with maximum lateness, windows accept late data
+