Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
2190c6d5
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
2190c6d5
编写于
11月 07, 2018
作者:
K
Kostas Kloudas
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[FLINK-10803][docs] Update the StreamingFileSink documentation for S3.
上级
16e05206
变更
1
显示空白变更内容
内联
并排
Showing
1 changed file
with
13 addition
and
4 deletion
+13
-4
docs/dev/connectors/streamfile_sink.md
docs/dev/connectors/streamfile_sink.md
+13
-4
未找到文件。
docs/dev/connectors/streamfile_sink.md
浏览文件 @
2190c6d5
...
...
@@ -24,16 +24,25 @@ under the License.
-->
This connector provides a Sink that writes partitioned files to filesystems
supported by the Flink
`FileSystem`
abstraction. Since in streaming the input
is potentially infinite, the streaming file sink writes data into buckets. The
bucketing behaviour is configurable but a useful default is time-based
supported by the
[
Flink `FileSystem` abstraction
](
{{
site.baseurl}}/ops/filesystems.html).
<span
class=
"label label-danger"
>
Important Note
</span>
: For S3, the
`StreamingFileSink`
supports only the
[
Hadoop-based
](
https://hadoop.apache.org/
)
FileSystem implementation, not
the implementation based on
[
Presto
](
https://prestodb.io/
)
. In case your job uses the
`StreamingFileSink`
to write to S3 but you want to use the Presto-based one for checkpointing,
it is advised to use explicitly
*"s3a://"*
(for Hadoop) as the scheme for the target path of
the sink and
*"s3p://"*
for checkpointing (for Presto). Using
*"s3://"*
for both the sink
and checkpointing may lead to unpredictable behavior, as both implementations "listen" to that scheme.
Since in streaming the input is potentially infinite, the streaming file sink writes data
into buckets. The bucketing behaviour is configurable but a useful default is time-based
bucketing where we start writing a new bucket every hour and thus get
individual files that each contain a part of the infinite output stream.
Within a bucket, we further split the output into smaller part files based on a
rolling policy. This is useful to prevent individual bucket files from getting
too big. This is also configurable but the default policy rolls files based on
file size and a timeout,
i.e
if no new data was written to a part file.
file size and a timeout,
*i.e*
if no new data was written to a part file.
The
`StreamingFileSink`
supports both row-wise encoding formats and
bulk-encoding formats, such as
[
Apache Parquet
](
http://parquet.apache.org
)
.
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录