Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
110bba38
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
110bba38
编写于
7月 26, 2016
作者:
T
twalthr
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[FLINK-4103] [table] Add CsvTableSource docs and Java accessibility
上级
7e309eeb
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
99 addition
and
4 deletion
+99
-4
docs/apis/table.md
docs/apis/table.md
+83
-0
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/sources/CsvTableSource.scala
...a/org/apache/flink/api/table/sources/CsvTableSource.scala
+16
-4
未找到文件。
docs/apis/table.md
浏览文件 @
110bba38
...
...
@@ -252,6 +252,89 @@ tableEnvironment.registerTableSource("kafka-source", kafkaTableSource);
Table
result
=
tableEnvironment
.
ingest
(
"kafka-source"
);
```
#### CsvTableSource
The
`CsvTableSource`
is already included in
`flink-table`
without additional dependecies.
It can be configured with the following properties:
-
`path`
The path to the CSV file, required.
-
`fieldNames`
The names of the table fields, required.
-
`fieldTypes`
The types of the table fields, required.
-
`fieldDelim`
The field delimiter,
`","`
by default.
-
`rowDelim`
The row delimiter,
`"\n"`
by default.
-
`quoteCharacter`
An optional quote character for String values,
`null`
by default.
-
`ignoreFirstLine`
Flag to ignore the first line,
`false`
by default.
-
`ignoreComments`
An optional prefix to indicate comments,
`null`
by default.
-
`lenient`
Flag to skip records with parse error instead to fail,
`false`
by default.
You can create the source as follows:
<div
class=
"codetabs"
markdown=
"1"
>
<div
data-lang=
"java"
markdown=
"1"
>
{% highlight java %}
CsvTableSource csvTableSource = new CsvTableSource(
"/path/to/your/file.csv",
new String[] { "name", "id", "score", "comments" },
new TypeInformation
<
?
>
[] {
Types.STRING(),
Types.INT(),
Types.DOUBLE(),
Types.STRING()
},
"#", // fieldDelim
"$", // rowDelim
null, // quoteCharacter
true, // ignoreFirstLine
"%", // ignoreComments
false); // lenient
{% endhighlight %}
</div>
<div
data-lang=
"scala"
markdown=
"1"
>
{% highlight scala %}
val csvTableSource = new CsvTableSource(
"/path/to/your/file.csv",
Array("name", "id", "score", "comments"),
Array(
Types.STRING,
Types.INT,
Types.DOUBLE,
Types.STRING
),
fieldDelim = "#",
rowDelim = "$",
ignoreFirstLine = true,
ignoreComments = "%")
{% endhighlight %}
</div>
</div>
You can work with the Table as explained in the rest of the Table API guide in both stream and batch
`TableEnvironment`
s:
<div
class=
"codetabs"
markdown=
"1"
>
<div
data-lang=
"java"
markdown=
"1"
>
{% highlight java %}
tableEnvironment.registerTableSource("mycsv", csvTableSource);
Table streamTable = streamTableEnvironment.ingest("mycsv");
Table batchTable = batchTableEnvironment.scan("mycsv");
{% endhighlight %}
</div>
<div
data-lang=
"scala"
markdown=
"1"
>
{% highlight scala %}
tableEnvironment.registerTableSource("mycsv", csvTableSource)
val streamTable = streamTableEnvironment.ingest("mycsv")
val batchTable = batchTableEnvironment.scan("mycsv")
{% endhighlight %}
</div>
</div>
Table API
----------
The Table API provides methods to apply relational operations on DataSets and Datastreams both in Scala and Java.
...
...
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/sources/CsvTableSource.scala
浏览文件 @
110bba38
...
...
@@ -35,11 +35,11 @@ import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment
* @param path The path to the CSV file.
* @param fieldNames The names of the table fields.
* @param fieldTypes The types of the table fields.
* @param fieldDelim The field delimiter,
','
by default.
* @param rowDelim The row delimiter,
'\n'
by default.
* @param quoteCharacter An optional quote character for String values,
disabled
by default.
* @param fieldDelim The field delimiter,
","
by default.
* @param rowDelim The row delimiter,
"\n"
by default.
* @param quoteCharacter An optional quote character for String values,
null
by default.
* @param ignoreFirstLine Flag to ignore the first line, false by default.
* @param ignoreComments An optional prefix to indicate comments,
disabled
by default.
* @param ignoreComments An optional prefix to indicate comments,
null
by default.
* @param lenient Flag to skip records with parse error instead to fail, false by default.
*/
class
CsvTableSource
(
...
...
@@ -55,6 +55,18 @@ class CsvTableSource(
extends
BatchTableSource
[
Row
]
with
StreamTableSource
[
Row
]
{
/**
* A [[BatchTableSource]] and [[StreamTableSource]] for simple CSV files with a
* (logically) unlimited number of fields.
*
* @param path The path to the CSV file.
* @param fieldNames The names of the table fields.
* @param fieldTypes The types of the table fields.
*/
def
this
(
path
:
String
,
fieldNames
:
Array
[
String
],
fieldTypes
:
Array
[
TypeInformation
[
_
]])
=
this
(
path
,
fieldNames
,
fieldTypes
,
CsvInputFormat
.
DEFAULT_FIELD_DELIMITER
,
CsvInputFormat
.
DEFAULT_LINE_DELIMITER
,
null
,
false
,
null
,
false
)
if
(
fieldNames
.
length
!=
fieldTypes
.
length
)
{
throw
TableException
(
"Number of field names and field types must be equal."
)
}
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录