Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
加钱居士.
foot
提交
3a9a25c9
F
foot
项目概览
加钱居士.
/
foot
通知
3
Star
1
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
foot
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
3a9a25c9
编写于
3月 16, 2020
作者:
S
shi.zeyuan
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
1.xxx
上级
1ae4836f
变更
7
隐藏空白更改
内联
并排
Showing
7 changed file
with
34 addition
and
34 deletion
+34
-34
conf/app.ini
conf/app.ini
+2
-2
foot-spider/launch/FS101PortalApplication.go
foot-spider/launch/FS101PortalApplication.go
+2
-2
foot-spider/module/win007/proc/AsiaLastNewProcesser.go
foot-spider/module/win007/proc/AsiaLastNewProcesser.go
+6
-6
foot-spider/module/win007/proc/BaseFaceProcesser.go
foot-spider/module/win007/proc/BaseFaceProcesser.go
+6
-6
foot-spider/module/win007/proc/EuroLastProcesser.go
foot-spider/module/win007/proc/EuroLastProcesser.go
+6
-6
foot-spider/module/win007/proc/EuroTrackProcesser.go
foot-spider/module/win007/proc/EuroTrackProcesser.go
+6
-6
foot-spider/module/win007/proc/MatchHisProcesser.go
foot-spider/module/win007/proc/MatchHisProcesser.go
+6
-6
未找到文件。
conf/app.ini
浏览文件 @
3a9a25c9
[mysql]
#
url=root:Meta.123@tcp(mysql.io:3306)/foot?charset=utf8
url
=
root:awe.FOG5188@tcp(39.108.212.245:3306)/foot?charset=utf8
url
=
root:Meta.123@tcp(mysql.io:3306)/foot?charset=utf8
#
url=root:awe.FOG5188@tcp(39.108.212.245:3306)/foot?charset=utf8
maxIdle
=
10
maxConn
=
50
[cookies]
...
...
foot-spider/launch/FS101PortalApplication.go
浏览文件 @
3a9a25c9
...
...
@@ -79,11 +79,11 @@ func Spider_History() {
Spider_match_his
(
v
)
Spider_euroLast_his
(
v
)
time
.
Sleep
(
28
*
time
.
Minute
)
Spider_
euroHis
_his
(
v
)
Spider_
asiaLastNew
_his
(
v
)
time
.
Sleep
(
28
*
time
.
Minute
)
Spider_baseFace_his
(
v
)
time
.
Sleep
(
28
*
time
.
Minute
)
Spider_
asiaLastNew
_his
(
v
)
Spider_
euroHis
_his
(
v
)
time
.
Sleep
(
28
*
time
.
Minute
)
mysql
.
ShowSQL
(
false
)
}
...
...
foot-spider/module/win007/proc/AsiaLastNewProcesser.go
浏览文件 @
3a9a25c9
...
...
@@ -50,6 +50,12 @@ func (this *AsiaLastNewProcesser) Startup() {
for
i
,
v
:=
range
this
.
MatchLastList
{
if
!
this
.
SingleThread
&&
i
%
1000
==
0
{
//10000个比赛一个spider,一个赛季大概有30万场比赛,最多30spider
//先将前面的spider启动
newSpider
.
SetDownloader
(
down
.
NewMAsiaLastApiDownloader
())
newSpider
=
newSpider
.
AddPipeline
(
pipeline
.
NewPipelineConsole
())
newSpider
.
SetSleepTime
(
"rand"
,
1
,
300
)
newSpider
.
SetThreadnum
(
1
)
.
Run
()
processer
=
GetAsiaLastNewProcesser
()
processer
.
Setup
(
this
)
newSpider
=
spider
.
NewSpider
(
processer
,
"AsiaLastNewProcesser"
+
strconv
.
Itoa
(
i
))
...
...
@@ -65,12 +71,6 @@ func (this *AsiaLastNewProcesser) Startup() {
url
:=
strings
.
Replace
(
win007
.
WIN007_ASIAODD_NEW_URL_PATTERN
,
"${matchId}"
,
win007_id
,
1
)
newSpider
=
newSpider
.
AddUrl
(
url
,
"json"
)
if
!
this
.
SingleThread
&&
i
%
1000
==
0
{
//10000个比赛一个spider,一个赛季大概有30万场比赛,最多30spider
newSpider
.
SetDownloader
(
down
.
NewMAsiaLastApiDownloader
())
newSpider
=
newSpider
.
AddPipeline
(
pipeline
.
NewPipelineConsole
())
newSpider
.
SetSleepTime
(
"rand"
,
1
,
300
)
newSpider
.
SetThreadnum
(
1
)
.
Run
()
}
}
newSpider
.
SetDownloader
(
down
.
NewMAsiaLastApiDownloader
())
...
...
foot-spider/module/win007/proc/BaseFaceProcesser.go
浏览文件 @
3a9a25c9
...
...
@@ -55,6 +55,12 @@ func (this *BaseFaceProcesser) Startup() {
for
i
,
v
:=
range
this
.
MatchLastList
{
if
!
this
.
SingleThread
&&
i
%
1000
==
0
{
//10000个比赛一个spider,一个赛季大概有30万场比赛,最多30spider
//先将前面的spider启动
newSpider
.
SetDownloader
(
down
.
NewMWin007Downloader
())
newSpider
=
newSpider
.
AddPipeline
(
pipeline
.
NewPipelineConsole
())
newSpider
.
SetSleepTime
(
"rand"
,
1
,
300
)
newSpider
.
SetThreadnum
(
1
)
.
Run
()
processer
=
GetBaseFaceProcesser
()
processer
.
Setup
(
this
)
newSpider
=
spider
.
NewSpider
(
processer
,
"BaseFaceProcesser"
+
strconv
.
Itoa
(
i
))
...
...
@@ -70,12 +76,6 @@ func (this *BaseFaceProcesser) Startup() {
url
:=
strings
.
Replace
(
win007
.
WIN007_BASE_FACE_URL_PATTERN
,
"${matchId}"
,
win007_id
,
1
)
newSpider
=
newSpider
.
AddUrl
(
url
,
"html"
)
if
!
this
.
SingleThread
&&
i
%
1000
==
0
{
//10000个比赛一个spider,一个赛季大概有30万场比赛,最多30spider
newSpider
.
SetDownloader
(
down
.
NewMWin007Downloader
())
newSpider
=
newSpider
.
AddPipeline
(
pipeline
.
NewPipelineConsole
())
newSpider
.
SetSleepTime
(
"rand"
,
1
,
300
)
newSpider
.
SetThreadnum
(
1
)
.
Run
()
}
}
newSpider
.
SetDownloader
(
down
.
NewMWin007Downloader
())
...
...
foot-spider/module/win007/proc/EuroLastProcesser.go
浏览文件 @
3a9a25c9
...
...
@@ -57,6 +57,12 @@ func (this *EuroLastProcesser) Startup() {
for
i
,
v
:=
range
this
.
MatchLastList
{
if
!
this
.
SingleThread
&&
i
%
1000
==
0
{
//10000个比赛一个spider,一个赛季大概有30万场比赛,最多30spider
//先将前面的spider启动
newSpider
.
SetDownloader
(
down
.
NewMWin007Downloader
())
newSpider
=
newSpider
.
AddPipeline
(
pipeline
.
NewPipelineConsole
())
newSpider
.
SetSleepTime
(
"rand"
,
1
,
300
)
newSpider
.
SetThreadnum
(
1
)
.
Run
()
processer
=
GetEuroLastProcesser
()
processer
.
Setup
(
this
)
newSpider
=
spider
.
NewSpider
(
processer
,
"EuroLastProcesser"
+
strconv
.
Itoa
(
i
))
...
...
@@ -72,12 +78,6 @@ func (this *EuroLastProcesser) Startup() {
url
:=
strings
.
Replace
(
win007
.
WIN007_EUROODD_URL_PATTERN
,
"${matchId}"
,
win007_id
,
1
)
newSpider
=
newSpider
.
AddUrl
(
url
,
"html"
)
if
!
this
.
SingleThread
&&
i
%
1000
==
0
{
//10000个比赛一个spider,一个赛季大概有30万场比赛,最多30spider
newSpider
.
SetDownloader
(
down
.
NewMWin007Downloader
())
newSpider
=
newSpider
.
AddPipeline
(
pipeline
.
NewPipelineConsole
())
newSpider
.
SetSleepTime
(
"rand"
,
1
,
300
)
newSpider
.
SetThreadnum
(
1
)
.
Run
()
}
}
newSpider
.
SetDownloader
(
down
.
NewMWin007Downloader
())
...
...
foot-spider/module/win007/proc/EuroTrackProcesser.go
浏览文件 @
3a9a25c9
...
...
@@ -56,6 +56,12 @@ func (this *EuroTrackProcesser) Startup() {
for
i
,
v
:=
range
this
.
MatchLastList
{
if
!
this
.
SingleThread
&&
i
%
1000
==
0
{
//10000个比赛一个spider,一个赛季大概有30万场比赛,最多30spider
//先将前面的spider启动
newSpider
.
SetDownloader
(
down
.
NewMWin007Downloader
())
newSpider
=
newSpider
.
AddPipeline
(
pipeline
.
NewPipelineConsole
())
newSpider
.
SetSleepTime
(
"rand"
,
1
,
300
)
newSpider
.
SetThreadnum
(
1
)
.
Run
()
processer
=
GetEuroTrackProcesser
()
processer
.
Setup
(
this
)
newSpider
=
spider
.
NewSpider
(
processer
,
"EuroTrackProcesser"
+
strconv
.
Itoa
(
i
))
...
...
@@ -73,12 +79,6 @@ func (this *EuroTrackProcesser) Startup() {
url
:=
strings
.
Replace
(
base_url
,
"${cId}"
,
v
,
1
)
newSpider
=
newSpider
.
AddUrl
(
url
,
"html"
)
}
if
!
this
.
SingleThread
&&
i
%
1000
==
0
{
//10000个比赛一个spider,一个赛季大概有30万场比赛,最多30spider
newSpider
.
SetDownloader
(
down
.
NewMWin007Downloader
())
newSpider
=
newSpider
.
AddPipeline
(
pipeline
.
NewPipelineConsole
())
newSpider
.
SetSleepTime
(
"rand"
,
1
,
300
)
newSpider
.
SetThreadnum
(
1
)
.
Run
()
}
}
newSpider
.
SetDownloader
(
down
.
NewMWin007Downloader
())
...
...
foot-spider/module/win007/proc/MatchHisProcesser.go
浏览文件 @
3a9a25c9
...
...
@@ -67,6 +67,12 @@ func (this *MatchHisProcesser) Startup() {
for
i
,
v
:=
range
seasonList
{
if
i
%
10
==
0
{
//10个联赛一个spider,总数1000多个联赛,最多100spider
//先将前面的spider启动
newSpider
.
SetDownloader
(
down
.
NewMWin007Downloader
())
newSpider
=
newSpider
.
AddPipeline
(
pipeline
.
NewPipelineConsole
())
newSpider
.
SetSleepTime
(
"rand"
,
1
,
300
)
newSpider
.
SetThreadnum
(
1
)
.
Run
()
processer
=
GetMatchHisProcesser
()
processer
.
Setup
(
this
)
newSpider
=
spider
.
NewSpider
(
processer
,
"MatchHisProcesser"
+
strconv
.
Itoa
(
i
))
...
...
@@ -84,12 +90,6 @@ func (this *MatchHisProcesser) Startup() {
processer
.
SUrl_Season
[
round_url
]
=
v
newSpider
=
newSpider
.
AddUrl
(
round_url
,
"html"
)
}
if
i
%
10
==
0
{
//10个联赛一个spider,总数1000多个联赛,最多100spider
newSpider
.
SetDownloader
(
down
.
NewMWin007Downloader
())
newSpider
=
newSpider
.
AddPipeline
(
pipeline
.
NewPipelineConsole
())
newSpider
.
SetSleepTime
(
"rand"
,
1
,
300
)
newSpider
.
SetThreadnum
(
1
)
.
Run
()
}
}
newSpider
.
SetDownloader
(
down
.
NewMWin007Downloader
())
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录