Licensed to the Apache Software Foundation (ASF) under one
...
...
@@ -22,14 +22,14 @@ under the License.
<ahref="#top"></a>
Analysis programs in Flink are regular programs that implement transformations on data sets
DataSet programs in Flink are regular programs that implement transformations on data sets
(e.g., filtering, mapping, joining, grouping). The data sets are initially created from certain
sources (e.g., by reading files, or from local collections). Results are returned via sinks, which may for
example write the data to (distributed) files, or to standard output (for example the command line
terminal). Flink programs run in a variety of contexts, standalone, or embedded in other programs.
The execution can happen in a local JVM, or on clusters of many machines.
In order to create your own Flink program, we encourage you to start with the
In order to create your own Flink DataSet program, we encourage you to start with the
[program skeleton](#program-skeleton) and gradually add your own
[transformations](#transformations). The remaining sections act as references for additional
operations and advanced features.
...
...
@@ -221,7 +221,7 @@ Program Skeleton
<divclass="codetabs"markdown="1">
<divdata-lang="java"markdown="1">
As we already saw in the example, Flink programs look like regular Java
As we already saw in the example, Flink DataSet programs look like regular Java
programs with a `main()` method. Each program consists of the same basic parts:
1. Obtain an `ExecutionEnvironment`,
...
...
@@ -233,7 +233,7 @@ programs with a `main()` method. Each program consists of the same basic parts:
We will now give an overview of each of those steps, please refer to the respective sections for
more details. Note that all core classes of the Java API are found in the package {% gh_link /flink-java/src/main/java/org/apache/flink/api/java "org.apache.flink.api.java" %}.
The `ExecutionEnvironment` is the basis for all Flink programs. You can
The `ExecutionEnvironment` is the basis for all Flink DataSet programs. You can
obtain one using these static methods on class `ExecutionEnvironment`:
{% highlight java %}
...
...
@@ -253,7 +253,7 @@ Typically, you only need to use `getExecutionEnvironment()`, since this
will do the right thing depending on the context: if you are executing
your program inside an IDE or as a regular Java program it will create
a local environment that will execute your program on your local machine. If
you created a JAR file from you program, and invoke it through the [command line](cli.html)
you created a JAR file from your program, and invoke it through the [command line](cli.html)
or the [web interface](web_client.html),
the Flink cluster manager will execute your main method and `getExecutionEnvironment()` will return
an execution environment for executing your program on a cluster.
...
...
@@ -276,7 +276,7 @@ more information on data sources and input formats, please refer to
Once you have a DataSet you can apply transformations to create a new
DataSet which you can then write to a file, transform again, or
combine with other DataSets. You apply transformations by calling
methods on DataSet with your own custom transformation function. For example,
methods on DataSet with your own custom transformation functions. For example,
a map transformation looks like this:
{% highlight java %}
...
...
@@ -447,18 +447,14 @@ accessed from the `getLastJobExecutionResult()` method.
DataSet abstraction
---------------
The batch processing APIs of Flink are centered around the `DataSet` abstraction. A `DataSet` is only
an abstract representation of a set of data that can contain duplicates.
Also note that Flink is not always physically creating (materializing) each DataSet at runtime. This
depends on the used runtime, the configuration and optimizer decisions.
The Flink runtime does not need to always materialize the DataSets because it is using a streaming runtime model.
DataSets are only materialized to avoid distributed deadlocks (at points where the data flow graph branches out and joins again later) or if the execution mode has explicitly been set to a batched execution.
A `DataSet` is an abstract representation of a finite immutable collection of data of the same type that may contain duplicates.
When using Flink on Tez, all DataSets are materialized.
Note that Flink is not always physically creating (materializing) each DataSet at runtime. This
depends on the used runtime, the configuration and optimizer decisions. DataSets may be "streamed through"
operations during execution, as under the hood Flink uses a streaming data processing engine.
Some DataSets are materialized automatically to avoid distributed deadlocks (at points where the data flow graph branches
out and joins again later) or if the execution mode has explicitly been set to blocking execution.
[Back to top](#top)
...
...
@@ -466,7 +462,7 @@ When using Flink on Tez, all DataSets are materialized.
Lazy Evaluation
---------------
All Flink programs are executed lazily: When the program's main method is executed, the data loading
All Flink DataSet programs are executed lazily: When the program's main method is executed, the data loading
and transformations do not happen directly. Rather, each operation is created and added to the
program's plan. The operations are actually executed when the execution is explicitly triggered by
an `execute()` call on the ExecutionEnvironment object. Also, `collect()` and `print()` will trigger
...
...
@@ -1323,7 +1319,7 @@ data.map (new MyMapFunction());
#### Anonymous classes
You can pass a function as an anonmymous class:
You can pass a function as an anonymous class:
{% highlight java %}
data.map(new MapFunction<String,Integer> () {
public Integer map(String value) { return Integer.parseInt(value); }
...
...
@@ -1451,7 +1447,7 @@ for a complete example.
Data Types
----------
Flink places some restrictions on the type of elements that are used in DataSets and as results
Flink places some restrictions on the type of elements that are used in DataSets and in results
of transformations. The reason for this is that the system analyzes the types to determine
efficient execution strategies.
...
...
@@ -1473,7 +1469,7 @@ Tuples are composite types that contain a fixed number of fields with various ty
The Java API provides classes from `Tuple1` up to `Tuple25`. Every field of a tuple
can be an arbitrary Flink type including further tuples, resulting in nested tuples. Fields of a
tuple can be accessed directly using the field's name as `tuple.f4`, or using the generic getter method
`tuple.getField(int position)`. The field indicies start at 0. Note that this stands in contrast
`tuple.getField(int position)`. The field indices start at 0. Note that this stands in contrast
to the Scala tuples, but it is more consistent with Java's general indexing.
{% highlight java %}
...
...
@@ -2239,7 +2235,7 @@ result data. This section give some hints how to ease the development of Flink p
### Local Execution Environment
A `LocalEnvironment` starts a Flink system within the same JVM process it was created in. If you
start the LocalEnvironement from an IDE, you can set breakpoint in your code and easily debug your
start the LocalEnvironement from an IDE, you can set breakpoints in your code and easily debug your
program.
A LocalEnvironment is created and used as follows:
...
...
@@ -2957,7 +2953,7 @@ public static final class Tokenizer extends RichFlatMapFunction<String, Tuple2<S
[Back to top](#top)
Program Packaging & Distributed Execution
Program Packaging and Distributed Execution
-----------------------------------------
As described in the [program skeleton](#program-skeleton) section, Flink programs can be executed on
@@ -62,13 +61,10 @@ This documentation is for Apache Flink version {{ site.version }}, which is the
You can download the latest pre-built snapshot version from the [downloads]({{ site.download_url }}#latest) page of the [project website]({{ site.website_url }}).
The Scala API uses Scala {{ site.scala_version }}. Please make sure to use a compatible version.
<!--The Scala API uses Scala {{ site.scala_version }}. Please make sure to use a compatible version.
Basically, the Scala API uses Scala 2.10. But you can use the API with Scala 2.11. To use Flink with
The Scala API uses Scala 2.10, but you can use the API with Scala 2.11. To use Flink with
