Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
looyolo
scrapy
提交
c0f3ca19
S
scrapy
项目概览
looyolo
/
scrapy
与 Fork 源项目一致
从无法访问的项目Fork
通知
2
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
S
scrapy
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
前往新版Gitcode,体验更适合开发者的 AI 搜索 >>
提交
c0f3ca19
编写于
3月 12, 2021
作者:
A
Adrián Chaves
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
get_retry_request: add typing information
上级
dfd4ab2b
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
15 addition
and
13 deletion
+15
-13
scrapy/downloadermiddlewares/retry.py
scrapy/downloadermiddlewares/retry.py
+15
-13
未找到文件。
scrapy/downloadermiddlewares/retry.py
浏览文件 @
c0f3ca19
...
...
@@ -9,8 +9,8 @@ RETRY_HTTP_CODES - which HTTP response codes to retry
Failed pages are collected on the scraping process and rescheduled at the end,
once the spider has finished crawling all regular (non failed) pages.
"""
import
logging
from
inspect
import
isclass
from
logging
import
getLogger
,
Logger
from
typing
import
Optional
,
Union
from
twisted.internet
import
defer
from
twisted.internet.error
import
(
...
...
@@ -24,24 +24,26 @@ from twisted.internet.error import (
)
from
twisted.web.client
import
ResponseFailed
from
scrapy.exceptions
import
NotConfigured
from
scrapy.utils.response
import
response_status_message
from
scrapy.core.downloader.handlers.http11
import
TunnelError
from
scrapy.exceptions
import
NotConfigured
from
scrapy.http.request
import
Request
from
scrapy.spiders
import
Spider
from
scrapy.utils.python
import
global_object_name
from
scrapy.utils.response
import
response_status_message
retry_logger
=
logging
.
getLogger
(
__name__
)
retry_logger
=
getLogger
(
__name__
)
def
get_retry_request
(
request
,
request
:
Request
,
*
,
spider
,
reason
=
'unspecified'
,
max_retry_times
=
None
,
priority_adjust
=
None
,
logger
=
retry_logger
,
stats_base_key
=
'retry'
,
spider
:
Spider
,
reason
:
Union
[
str
,
Exception
]
=
'unspecified'
,
max_retry_times
:
Optional
[
int
]
=
None
,
priority_adjust
:
Union
[
int
,
float
,
None
]
=
None
,
logger
:
Logger
=
retry_logger
,
stats_base_key
:
str
=
'retry'
,
):
"""
Returns a new :class:`~scrapy.Request` object to retry the specified
...
...
@@ -105,7 +107,7 @@ def get_retry_request(
priority_adjust
=
settings
.
getint
(
'RETRY_PRIORITY_ADJUST'
)
new_request
.
priority
=
request
.
priority
+
priority_adjust
if
isclass
(
reason
):
if
callable
(
reason
):
reason
=
reason
()
if
isinstance
(
reason
,
Exception
):
reason
=
global_object_name
(
reason
.
__class__
)
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录