Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
looyolo
scrapy
提交
ad14cf05
S
scrapy
项目概览
looyolo
/
scrapy
与 Fork 源项目一致
从无法访问的项目Fork
通知
2
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
S
scrapy
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
ad14cf05
编写于
4月 28, 2014
作者:
D
Daniel Graña
浏览文件
操作
浏览文件
下载
差异文件
Merge pull request #597 from chekunkov/dupefilter_request_fingerprint_test_and_docs
Dupefilter request_fingerprint test and docs
上级
3932877b
4aa6179a
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
45 addition
and
9 deletion
+45
-9
docs/topics/settings.rst
docs/topics/settings.rst
+5
-1
scrapy/tests/test_dupefilter.py
scrapy/tests/test_dupefilter.py
+40
-8
未找到文件。
docs/topics/settings.rst
浏览文件 @
ad14cf05
...
...
@@ -408,7 +408,11 @@ Default: ``'scrapy.dupefilter.RFPDupeFilter'``
The class used to detect and filter duplicate requests.
The default (``RFPDupeFilter``) filters based on request fingerprint using
the ``scrapy.utils.request.request_fingerprint`` function.
the ``scrapy.utils.request.request_fingerprint`` function. In order to change
the way duplicates are checked you could subclass ``RFPDupeFilter`` and
override its ``request_fingerprint`` method. This method should accept
scrapy :class:`~scrapy.http.Request` object and return its fingerprint
(a string).
.. setting:: DUPEFILTER_DEBUG
...
...
scrapy/tests/test_dupefilter.py
浏览文件 @
ad14cf05
import
hashlib
import
unittest
from
scrapy.http
import
Request
from
scrapy.dupefilter
import
RFPDupeFilter
from
scrapy.http
import
Request
class
RFPDupeFilterTest
(
unittest
.
TestCase
):
def
test_filter
(
self
):
filter
=
RFPDupeFilter
()
filter
.
open
()
dupe
filter
=
RFPDupeFilter
()
dupe
filter
.
open
()
r1
=
Request
(
'http://scrapytest.org/1'
)
r2
=
Request
(
'http://scrapytest.org/2'
)
r3
=
Request
(
'http://scrapytest.org/2'
)
assert
not
filter
.
request_seen
(
r1
)
assert
filter
.
request_seen
(
r1
)
assert
not
dupefilter
.
request_seen
(
r1
)
assert
dupefilter
.
request_seen
(
r1
)
assert
not
dupefilter
.
request_seen
(
r2
)
assert
dupefilter
.
request_seen
(
r3
)
dupefilter
.
close
(
'finished'
)
def
test_request_fingerprint
(
self
):
"""Test if customization of request_fingerprint method will change
output of request_seen.
"""
r1
=
Request
(
'http://scrapytest.org/index.html'
)
r2
=
Request
(
'http://scrapytest.org/INDEX.html'
)
dupefilter
=
RFPDupeFilter
()
dupefilter
.
open
()
assert
not
dupefilter
.
request_seen
(
r1
)
assert
not
dupefilter
.
request_seen
(
r2
)
dupefilter
.
close
(
'finished'
)
class
CaseInsensitiveRFPDupeFilter
(
RFPDupeFilter
):
def
request_fingerprint
(
self
,
request
):
fp
=
hashlib
.
sha1
()
fp
.
update
(
request
.
url
.
lower
())
return
fp
.
hexdigest
()
case_insensitive_dupefilter
=
CaseInsensitiveRFPDupeFilter
()
case_insensitive_dupefilter
.
open
()
assert
not
filter
.
request_seen
(
r2
)
assert
filter
.
request_seen
(
r3
)
assert
not
case_insensitive_dupefilter
.
request_seen
(
r1
)
assert
case_insensitive_dupefilter
.
request_seen
(
r2
)
filter
.
close
(
'finished'
)
case_insensitive_dupe
filter
.
close
(
'finished'
)
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录