Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
ssssssss-team
spider-flow
提交
6c654211
S
spider-flow
项目概览
ssssssss-team
/
spider-flow
10 个月 前同步成功
通知
0
Star
1
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
S
spider-flow
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
前往新版Gitcode,体验更适合开发者的 AI 搜索 >>
未验证
提交
6c654211
编写于
4月 11, 2020
作者:
J
javamxd
提交者:
GitHub
4月 11, 2020
浏览文件
操作
浏览文件
下载
差异文件
Merge pull request #2 from nekolr/dev
修复输出 csv 文件乱码的问题
上级
ed4ec48d
6543a74c
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
54 addition
and
24 deletion
+54
-24
spider-flow-core/src/main/java/org/spiderflow/core/executor/shape/OutputExecutor.java
...va/org/spiderflow/core/executor/shape/OutputExecutor.java
+45
-24
spider-flow-web/src/main/resources/static/resources/templates/output.html
...src/main/resources/static/resources/templates/output.html
+9
-0
未找到文件。
spider-flow-core/src/main/java/org/spiderflow/core/executor/shape/OutputExecutor.java
浏览文件 @
6c654211
...
...
@@ -14,6 +14,7 @@ import org.spiderflow.core.utils.DataSourceUtils;
import
org.spiderflow.core.utils.ExpressionUtils
;
import
org.spiderflow.executor.ShapeExecutor
;
import
org.spiderflow.io.SpiderResponse
;
import
org.spiderflow.listener.SpiderListener
;
import
org.spiderflow.model.SpiderNode
;
import
org.spiderflow.model.SpiderOutput
;
import
org.springframework.jdbc.core.JdbcTemplate
;
...
...
@@ -28,12 +29,12 @@ import java.util.*;
*
*/
@Component
public
class
OutputExecutor
implements
ShapeExecutor
{
public
class
OutputExecutor
implements
ShapeExecutor
,
SpiderListener
{
public
static
final
String
OUTPUT_ALL
=
"output-all"
;
public
static
final
String
OUTPUT_NAME
=
"output-name"
;
public
static
final
String
OUTPUT_VALUE
=
"output-value"
;
public
static
final
String
DATASOURCE_ID
=
"datasourceId"
;
...
...
@@ -46,13 +47,15 @@ public class OutputExecutor implements ShapeExecutor{
public
static
final
String
CSV_NAME
=
"csvName"
;
public
static
final
String
CSV_ENCODING
=
"csvEncoding"
;
private
static
Logger
logger
=
LoggerFactory
.
getLogger
(
OutputExecutor
.
class
);
/**
* 输出CSVPrinter节点变量
*/
private
Map
<
String
,
CSVPrinter
>
cachePrinter
=
new
HashMap
<>();
@Override
public
void
execute
(
SpiderNode
node
,
SpiderContext
context
,
Map
<
String
,
Object
>
variables
)
{
SpiderOutput
output
=
new
SpiderOutput
();
...
...
@@ -103,24 +106,6 @@ public class OutputExecutor implements ShapeExecutor{
context
.
addOutput
(
output
);
}
@Override
public
boolean
allowExecuteNext
(
SpiderNode
node
,
SpiderContext
context
,
Map
<
String
,
Object
>
variables
)
{
String
key
=
context
.
getId
()
+
"-"
+
node
.
getNodeId
();
if
(
node
.
isDone
()){
CSVPrinter
printer
=
cachePrinter
.
remove
(
key
);
if
(
printer
!=
null
){
try
{
printer
.
flush
();
printer
.
close
();
}
catch
(
IOException
e
)
{
logger
.
error
(
"文件输出错误,异常信息:{}"
,
e
.
getMessage
(),
e
);
ExceptionUtils
.
wrapAndThrow
(
e
);
}
}
}
return
true
;
}
/**
* 输出所有参数
* @param output
...
...
@@ -187,14 +172,24 @@ public class OutputExecutor implements ShapeExecutor{
try
{
if
(
printer
==
null
)
{
CSVFormat
format
=
CSVFormat
.
DEFAULT
.
withHeader
(
headers
);
OutputStreamWriter
osw
=
new
OutputStreamWriter
(
new
FileOutputStream
(
csvName
),
"UTF-8"
);
FileOutputStream
os
=
new
FileOutputStream
(
csvName
);
String
csvEncoding
=
node
.
getStringJsonValue
(
CSV_ENCODING
);
if
(
"UTF-8BOM"
.
equals
(
csvEncoding
))
{
csvEncoding
=
csvEncoding
.
substring
(
0
,
5
);
byte
[]
bom
=
{(
byte
)
0xEF
,
(
byte
)
0xBB
,
(
byte
)
0xBF
};
os
.
write
(
bom
);
os
.
flush
();
}
OutputStreamWriter
osw
=
new
OutputStreamWriter
(
os
,
csvEncoding
);
printer
=
new
CSVPrinter
(
osw
,
format
);
cachePrinter
.
put
(
key
,
printer
);
}
for
(
int
i
=
0
;
i
<
headers
.
length
;
i
++)
{
records
.
add
(
data
.
get
(
headers
[
i
]).
toString
());
}
printer
.
printRecord
(
records
);
synchronized
(
printer
)
{
printer
.
printRecord
(
records
);
}
}
catch
(
IOException
e
)
{
logger
.
error
(
"文件输出错误,异常信息:{}"
,
e
.
getMessage
(),
e
);
ExceptionUtils
.
wrapAndThrow
(
e
);
...
...
@@ -206,4 +201,30 @@ public class OutputExecutor implements ShapeExecutor{
return
"output"
;
}
@Override
public
void
beforeStart
(
SpiderContext
context
)
{
}
@Override
public
void
afterEnd
(
SpiderContext
context
)
{
this
.
releasePrinters
();
}
private
void
releasePrinters
()
{
for
(
Iterator
<
Map
.
Entry
<
String
,
CSVPrinter
>>
iterator
=
this
.
cachePrinter
.
entrySet
().
iterator
();
iterator
.
hasNext
();
)
{
Map
.
Entry
<
String
,
CSVPrinter
>
entry
=
iterator
.
next
();
CSVPrinter
printer
=
entry
.
getValue
();
if
(
printer
!=
null
)
{
try
{
printer
.
flush
();
printer
.
close
();
this
.
cachePrinter
.
remove
(
entry
.
getKey
());
}
catch
(
IOException
e
)
{
logger
.
error
(
"文件输出错误,异常信息:{}"
,
e
.
getMessage
(),
e
);
ExceptionUtils
.
wrapAndThrow
(
e
);
}
}
}
}
}
spider-flow-web/src/main/resources/static/resources/templates/output.html
浏览文件 @
6c654211
...
...
@@ -69,6 +69,15 @@
<input
type=
"text"
name=
"csvName"
placeholder=
"请输入文件名"
autocomplete=
"off"
class=
"layui-input input-default"
value=
"{{=d.data.object.csvName}}"
>
</div>
</div>
<div
class=
"layui-col-md2 csvDiv"
{{
d.data.object
['
output-csv
'
] =
=
'1'
?
''
:
'
style=
"display: none;"
'}}
>
<div
class=
"layui-form-item"
>
<select
name=
"csvEncoding"
>
<option
value=
"GBK"
{{
d.data.object
['
csvEncoding
'
] =
=
'
GBK
'
?
'
selected
'
:
''}}
>
GBK
</option>
<option
value=
"UTF-8"
{{
d.data.object
['
csvEncoding
'
] =
=
'
UTF-8
'
?
'
selected
'
:
''}}
>
UTF-8
</option>
<option
value=
"UTF-8BOM"
{{
d.data.object
['
csvEncoding
'
] =
=
'
UTF-8BOM
'
?
'
selected
'
:
''}}
>
UTF-8 BOM
</option>
</select>
</div>
</div>
</div>
<table
class=
"layui-table"
id=
"spider-output"
data-cell=
"{{=d.cell.id}}"
data-keys=
"output-name,output-value"
></table>
<hr>
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录