Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
looyolo
scrapy
提交
d684ecad
S
scrapy
项目概览
looyolo
/
scrapy
与 Fork 源项目一致
从无法访问的项目Fork
通知
2
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
S
scrapy
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
d684ecad
编写于
8月 18, 2014
作者:
D
Daniel Graña
浏览文件
操作
浏览文件
下载
差异文件
Merge pull request #846 from rocioar/master
fix dont_merge_cookies bad behaviour when set to false on meta
上级
94d00b2a
51b0bd28
变更
8
隐藏空白更改
内联
并排
Showing
8 changed file
with
36 addition
and
13 deletion
+36
-13
docs/topics/downloader-middleware.rst
docs/topics/downloader-middleware.rst
+4
-4
docs/topics/request-response.rst
docs/topics/request-response.rst
+2
-2
scrapy/contrib/downloadermiddleware/cookies.py
scrapy/contrib/downloadermiddleware/cookies.py
+2
-2
scrapy/contrib/downloadermiddleware/redirect.py
scrapy/contrib/downloadermiddleware/redirect.py
+2
-2
scrapy/contrib/downloadermiddleware/retry.py
scrapy/contrib/downloadermiddleware/retry.py
+3
-3
tests/test_downloadermiddleware_cookies.py
tests/test_downloadermiddleware_cookies.py
+6
-0
tests/test_downloadermiddleware_redirect.py
tests/test_downloadermiddleware_redirect.py
+9
-0
tests/test_downloadermiddleware_retry.py
tests/test_downloadermiddleware_retry.py
+8
-0
未找到文件。
docs/topics/downloader-middleware.rst
浏览文件 @
d684ecad
...
...
@@ -634,8 +634,8 @@ settings (see the settings documentation for more info):
.. reqmeta:: dont_redirect
If :attr:`Request.meta <scrapy.http.Request.meta>`
contains the
``dont_redirect`` key
, the request will be ignored by this middleware.
If :attr:`Request.meta <scrapy.http.Request.meta>`
has ``dont_redirect``
key set to True
, the request will be ignored by this middleware.
RedirectMiddleware settings
...
...
@@ -732,8 +732,8 @@ to indicate server overload, which would be something we want to retry.
.. reqmeta:: dont_retry
If :attr:`Request.meta <scrapy.http.Request.meta>`
contains the ``dont_retry``
key
, the request will be ignored by this middleware.
If :attr:`Request.meta <scrapy.http.Request.meta>`
has ``dont_retry`` key
set to True
, the request will be ignored by this middleware.
RetryMiddleware Settings
~~~~~~~~~~~~~~~~~~~~~~~~
...
...
docs/topics/request-response.rst
浏览文件 @
d684ecad
...
...
@@ -83,7 +83,7 @@ Request objects
cookies for that domain and will be sent again in future requests. That's
the typical behaviour of any regular web browser. However, if, for some
reason, you want to avoid merging with existing cookies you can instruct
Scrapy to do so by setting the ``dont_merge_cookies`` key in the
Scrapy to do so by setting the ``dont_merge_cookies`` key
to True
in the
:attr:`Request.meta`.
Example of request without merging cookies::
...
...
@@ -102,7 +102,7 @@ Request objects
:param priority: the priority of this request (defaults to ``0``).
The priority is used by the scheduler to define the order used to process
requests. Requests with a higher priority value will execute earlier.
requests. Requests with a higher priority value will execute earlier.
Negative values are allowed in order to indicate relatively low-priority.
:type priority: int
...
...
scrapy/contrib/downloadermiddleware/cookies.py
浏览文件 @
d684ecad
...
...
@@ -22,7 +22,7 @@ class CookiesMiddleware(object):
return
cls
(
crawler
.
settings
.
getbool
(
'COOKIES_DEBUG'
))
def
process_request
(
self
,
request
,
spider
):
if
'dont_merge_cookies'
in
request
.
meta
:
if
request
.
meta
.
get
(
'dont_merge_cookies'
,
False
)
:
return
cookiejarkey
=
request
.
meta
.
get
(
"cookiejar"
)
...
...
@@ -37,7 +37,7 @@ class CookiesMiddleware(object):
self
.
_debug_cookie
(
request
,
spider
)
def
process_response
(
self
,
request
,
response
,
spider
):
if
'dont_merge_cookies'
in
request
.
meta
:
if
request
.
meta
.
get
(
'dont_merge_cookies'
,
False
)
:
return
response
# extract cookies from Set-Cookie and drop invalid/expired cookies
...
...
scrapy/contrib/downloadermiddleware/redirect.py
浏览文件 @
d684ecad
...
...
@@ -52,7 +52,7 @@ class RedirectMiddleware(BaseRedirectMiddleware):
"""Handle redirection of requests based on response status and meta-refresh html tag"""
def
process_response
(
self
,
request
,
response
,
spider
):
if
'dont_redirect'
in
request
.
meta
:
if
request
.
meta
.
get
(
'dont_redirect'
,
False
)
:
return
response
if
request
.
method
==
'HEAD'
:
...
...
@@ -86,7 +86,7 @@ class MetaRefreshMiddleware(BaseRedirectMiddleware):
settings
.
getint
(
'METAREFRESH_MAXDELAY'
))
def
process_response
(
self
,
request
,
response
,
spider
):
if
'dont_redirect'
in
request
.
meta
or
request
.
method
==
'HEAD'
or
\
if
request
.
meta
.
get
(
'dont_redirect'
,
False
)
or
request
.
method
==
'HEAD'
or
\
not
isinstance
(
response
,
HtmlResponse
):
return
response
...
...
scrapy/contrib/downloadermiddleware/retry.py
浏览文件 @
d684ecad
...
...
@@ -50,7 +50,7 @@ class RetryMiddleware(object):
return
cls
(
crawler
.
settings
)
def
process_response
(
self
,
request
,
response
,
spider
):
if
'dont_retry'
in
request
.
meta
:
if
request
.
meta
.
get
(
'dont_retry'
,
False
)
:
return
response
if
response
.
status
in
self
.
retry_http_codes
:
reason
=
response_status_message
(
response
.
status
)
...
...
@@ -59,8 +59,8 @@ class RetryMiddleware(object):
def
process_exception
(
self
,
request
,
exception
,
spider
):
if
isinstance
(
exception
,
self
.
EXCEPTIONS_TO_RETRY
)
\
and
'dont_retry'
not
in
request
.
meta
:
return
self
.
_retry
(
request
,
exception
,
spider
)
and
not
request
.
meta
.
get
(
'dont_retry'
,
False
)
:
return
self
.
_retry
(
request
,
exception
,
spider
)
def
_retry
(
self
,
request
,
reason
,
spider
):
retries
=
request
.
meta
.
get
(
'retry_times'
,
0
)
+
1
...
...
tests/test_downloadermiddleware_cookies.py
浏览文件 @
d684ecad
...
...
@@ -52,10 +52,16 @@ class CookiesMiddlewareTest(TestCase):
res
=
Response
(
'http://scrapytest.org/dontmerge'
,
headers
=
{
'Set-Cookie'
:
'dont=mergeme; path=/'
})
assert
self
.
mw
.
process_response
(
req
,
res
,
self
.
spider
)
is
res
# check that cookies are merged back
req
=
Request
(
'http://scrapytest.org/mergeme'
)
assert
self
.
mw
.
process_request
(
req
,
self
.
spider
)
is
None
self
.
assertEquals
(
req
.
headers
.
get
(
'Cookie'
),
'C1=value1'
)
# check that cookies are merged when dont_merge_cookies is passed as 0
req
=
Request
(
'http://scrapytest.org/mergeme'
,
meta
=
{
'dont_merge_cookies'
:
0
})
assert
self
.
mw
.
process_request
(
req
,
self
.
spider
)
is
None
self
.
assertEquals
(
req
.
headers
.
get
(
'Cookie'
),
'C1=value1'
)
def
test_complex_cookies
(
self
):
# merge some cookies into jar
cookies
=
[{
'name'
:
'C1'
,
'value'
:
'value1'
,
'path'
:
'/foo'
,
'domain'
:
'scrapytest.org'
},
...
...
tests/test_downloadermiddleware_redirect.py
浏览文件 @
d684ecad
...
...
@@ -50,6 +50,15 @@ class RedirectMiddlewareTest(unittest.TestCase):
assert
isinstance
(
r
,
Response
)
assert
r
is
rsp
# Test that it redirects when dont_redirect is False
req
=
Request
(
url
,
meta
=
{
'dont_redirect'
:
False
})
rsp
=
Response
(
url2
,
status
=
200
)
r
=
self
.
mw
.
process_response
(
req
,
rsp
,
self
.
spider
)
assert
isinstance
(
r
,
Response
)
assert
r
is
rsp
def
test_redirect_302
(
self
):
url
=
'http://www.example.com/302'
url2
=
'http://www.example.com/redirected2'
...
...
tests/test_downloadermiddleware_retry.py
浏览文件 @
d684ecad
...
...
@@ -40,6 +40,14 @@ class RetryTest(unittest.TestCase):
r
=
self
.
mw
.
process_response
(
req
,
rsp
,
self
.
spider
)
assert
r
is
rsp
# Test retry when dont_retry set to False
req
=
Request
(
'http://www.scrapytest.org/503'
,
meta
=
{
'dont_retry'
:
False
})
rsp
=
Response
(
'http://www.scrapytest.org/503'
)
# first retry
r
=
self
.
mw
.
process_response
(
req
,
rsp
,
self
.
spider
)
assert
r
is
rsp
def
test_dont_retry_exc
(
self
):
req
=
Request
(
'http://www.scrapytest.org/503'
,
meta
=
{
'dont_retry'
:
True
})
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录