Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
looyolo
scrapy
提交
180272c0
S
scrapy
项目概览
looyolo
/
scrapy
与 Fork 源项目一致
从无法访问的项目Fork
通知
2
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
S
scrapy
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
180272c0
编写于
4月 21, 2015
作者:
J
Julia Medina
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Move scrapy/contrib/spidermiddleware to scrapy/spidermiddlewares
上级
c97a69c9
变更
16
隐藏空白更改
内联
并排
Showing
16 changed file
with
26 addition
and
26 deletion
+26
-26
docs/faq.rst
docs/faq.rst
+1
-1
docs/topics/settings.rst
docs/topics/settings.rst
+6
-6
docs/topics/spider-middleware.rst
docs/topics/spider-middleware.rst
+7
-7
docs/topics/spiders.rst
docs/topics/spiders.rst
+1
-1
scrapy/settings/default_settings.py
scrapy/settings/default_settings.py
+5
-5
scrapy/spidermiddlewares/__init__.py
scrapy/spidermiddlewares/__init__.py
+0
-0
scrapy/spidermiddlewares/depth.py
scrapy/spidermiddlewares/depth.py
+0
-0
scrapy/spidermiddlewares/httperror.py
scrapy/spidermiddlewares/httperror.py
+0
-0
scrapy/spidermiddlewares/offsite.py
scrapy/spidermiddlewares/offsite.py
+0
-0
scrapy/spidermiddlewares/referer.py
scrapy/spidermiddlewares/referer.py
+0
-0
scrapy/spidermiddlewares/urllength.py
scrapy/spidermiddlewares/urllength.py
+0
-0
tests/test_spidermiddleware_depth.py
tests/test_spidermiddleware_depth.py
+2
-2
tests/test_spidermiddleware_httperror.py
tests/test_spidermiddleware_httperror.py
+1
-1
tests/test_spidermiddleware_offsite.py
tests/test_spidermiddleware_offsite.py
+1
-1
tests/test_spidermiddleware_referer.py
tests/test_spidermiddleware_referer.py
+1
-1
tests/test_spidermiddleware_urllength.py
tests/test_spidermiddleware_urllength.py
+1
-1
未找到文件。
docs/faq.rst
浏览文件 @
180272c0
...
...
@@ -149,7 +149,7 @@ middleware (enabled by default) whose purpose is to filter out requests to
domains outside the ones covered by the spider.
For more info see:
:class:`~scrapy.
contrib.spidermiddleware
.offsite.OffsiteMiddleware`.
:class:`~scrapy.
spidermiddlewares
.offsite.OffsiteMiddleware`.
What is the recommended way to deploy a Scrapy crawler in production?
---------------------------------------------------------------------
...
...
docs/topics/settings.rst
浏览文件 @
180272c0
...
...
@@ -906,11 +906,11 @@ SPIDER_MIDDLEWARES_BASE
Default::
{
'scrapy.
contrib.spidermiddleware
.httperror.HttpErrorMiddleware': 50,
'scrapy.
contrib.spidermiddleware
.offsite.OffsiteMiddleware': 500,
'scrapy.
contrib.spidermiddleware
.referer.RefererMiddleware': 700,
'scrapy.
contrib.spidermiddleware
.urllength.UrlLengthMiddleware': 800,
'scrapy.
contrib.spidermiddleware
.depth.DepthMiddleware': 900,
'scrapy.
spidermiddlewares
.httperror.HttpErrorMiddleware': 50,
'scrapy.
spidermiddlewares
.offsite.OffsiteMiddleware': 500,
'scrapy.
spidermiddlewares
.referer.RefererMiddleware': 700,
'scrapy.
spidermiddlewares
.urllength.UrlLengthMiddleware': 800,
'scrapy.
spidermiddlewares
.depth.DepthMiddleware': 900,
}
A dict containing the spider middlewares enabled by default in Scrapy. You
...
...
@@ -1001,7 +1001,7 @@ URLLENGTH_LIMIT
Default: ``2083``
Scope: ``
contrib.spidermiddleware
.urllength``
Scope: ``
spidermiddlewares
.urllength``
The maximum URL length to allow for crawled URLs. For more information about
the default value for this setting see: http://www.boutell.com/newfaq/misc/urllength.html
...
...
docs/topics/spider-middleware.rst
浏览文件 @
180272c0
...
...
@@ -43,7 +43,7 @@ value. For example, if you want to disable the off-site middleware::
SPIDER_MIDDLEWARES = {
'myproject.middlewares.CustomSpiderMiddleware': 543,
'scrapy.
contrib.spidermiddleware
.offsite.OffsiteMiddleware': None,
'scrapy.
spidermiddlewares
.offsite.OffsiteMiddleware': None,
}
Finally, keep in mind that some middlewares may need to be enabled through a
...
...
@@ -55,7 +55,7 @@ Writing your own spider middleware
Each middleware component is a Python class that defines one or more of the
following methods:
.. module:: scrapy.
contrib.spidermiddleware
.. module:: scrapy.
spidermiddlewares
.. class:: SpiderMiddleware
...
...
@@ -178,7 +178,7 @@ For a list of the components enabled by default (and their orders) see the
DepthMiddleware
---------------
.. module:: scrapy.
contrib.spidermiddleware
.depth
.. module:: scrapy.
spidermiddlewares
.depth
:synopsis: Depth Spider Middleware
.. class:: DepthMiddleware
...
...
@@ -199,7 +199,7 @@ DepthMiddleware
HttpErrorMiddleware
-------------------
.. module:: scrapy.
contrib.spidermiddleware
.httperror
.. module:: scrapy.
spidermiddlewares
.httperror
:synopsis: HTTP Error Spider Middleware
.. class:: HttpErrorMiddleware
...
...
@@ -264,7 +264,7 @@ Pass all responses, regardless of its status code.
OffsiteMiddleware
-----------------
.. module:: scrapy.
contrib.spidermiddleware
.offsite
.. module:: scrapy.
spidermiddlewares
.offsite
:synopsis: Offsite Spider Middleware
.. class:: OffsiteMiddleware
...
...
@@ -298,7 +298,7 @@ OffsiteMiddleware
RefererMiddleware
-----------------
.. module:: scrapy.
contrib.spidermiddleware
.referer
.. module:: scrapy.
spidermiddlewares
.referer
:synopsis: Referer Spider Middleware
.. class:: RefererMiddleware
...
...
@@ -323,7 +323,7 @@ Whether to enable referer middleware.
UrlLengthMiddleware
-------------------
.. module:: scrapy.
contrib.spidermiddleware
.urllength
.. module:: scrapy.
spidermiddlewares
.urllength
:synopsis: URL Length Spider Middleware
.. class:: UrlLengthMiddleware
...
...
docs/topics/spiders.rst
浏览文件 @
180272c0
...
...
@@ -77,7 +77,7 @@ scrapy.Spider
An optional list of strings containing domains that this spider is
allowed to crawl. Requests for URLs not belonging to the domain names
specified in this list won't be followed if
:class:`~scrapy.
contrib.spidermiddleware
.offsite.OffsiteMiddleware` is enabled.
:class:`~scrapy.
spidermiddlewares
.offsite.OffsiteMiddleware` is enabled.
.. attribute:: start_urls
...
...
scrapy/settings/default_settings.py
浏览文件 @
180272c0
...
...
@@ -224,11 +224,11 @@ SPIDER_MIDDLEWARES = {}
SPIDER_MIDDLEWARES_BASE
=
{
# Engine side
'scrapy.
contrib.spidermiddleware
.httperror.HttpErrorMiddleware'
:
50
,
'scrapy.
contrib.spidermiddleware
.offsite.OffsiteMiddleware'
:
500
,
'scrapy.
contrib.spidermiddleware
.referer.RefererMiddleware'
:
700
,
'scrapy.
contrib.spidermiddleware
.urllength.UrlLengthMiddleware'
:
800
,
'scrapy.
contrib.spidermiddleware
.depth.DepthMiddleware'
:
900
,
'scrapy.
spidermiddlewares
.httperror.HttpErrorMiddleware'
:
50
,
'scrapy.
spidermiddlewares
.offsite.OffsiteMiddleware'
:
500
,
'scrapy.
spidermiddlewares
.referer.RefererMiddleware'
:
700
,
'scrapy.
spidermiddlewares
.urllength.UrlLengthMiddleware'
:
800
,
'scrapy.
spidermiddlewares
.depth.DepthMiddleware'
:
900
,
# Spider side
}
...
...
scrapy/
contrib/spidermiddleware
/__init__.py
→
scrapy/
spidermiddlewares
/__init__.py
浏览文件 @
180272c0
文件已移动
scrapy/
contrib/spidermiddleware
/depth.py
→
scrapy/
spidermiddlewares
/depth.py
浏览文件 @
180272c0
文件已移动
scrapy/
contrib/spidermiddleware
/httperror.py
→
scrapy/
spidermiddlewares
/httperror.py
浏览文件 @
180272c0
文件已移动
scrapy/
contrib/spidermiddleware
/offsite.py
→
scrapy/
spidermiddlewares
/offsite.py
浏览文件 @
180272c0
文件已移动
scrapy/
contrib/spidermiddleware
/referer.py
→
scrapy/
spidermiddlewares
/referer.py
浏览文件 @
180272c0
文件已移动
scrapy/
contrib/spidermiddleware
/urllength.py
→
scrapy/
spidermiddlewares
/urllength.py
浏览文件 @
180272c0
文件已移动
tests/test_spidermiddleware_depth.py
浏览文件 @
180272c0
from
unittest
import
TestCase
from
scrapy.
contrib.spidermiddleware
.depth
import
DepthMiddleware
from
scrapy.
spidermiddlewares
.depth
import
DepthMiddleware
from
scrapy.http
import
Response
,
Request
from
scrapy.spider
import
Spider
from
scrapy.statscol
import
StatsCollector
...
...
@@ -37,7 +37,7 @@ class TestDepthMiddleware(TestCase):
rdm
=
self
.
stats
.
get_value
(
'request_depth_max'
,
spider
=
self
.
spider
)
self
.
assertEquals
(
rdm
,
1
)
def
tearDown
(
self
):
self
.
stats
.
close_spider
(
self
.
spider
,
''
)
tests/test_spidermiddleware_httperror.py
浏览文件 @
180272c0
...
...
@@ -8,7 +8,7 @@ from scrapy.utils.test import get_crawler
from
tests.mockserver
import
MockServer
from
scrapy.http
import
Response
,
Request
from
scrapy.spider
import
Spider
from
scrapy.
contrib.spidermiddleware
.httperror
import
HttpErrorMiddleware
,
HttpError
from
scrapy.
spidermiddlewares
.httperror
import
HttpErrorMiddleware
,
HttpError
from
scrapy.settings
import
Settings
...
...
tests/test_spidermiddleware_offsite.py
浏览文件 @
180272c0
...
...
@@ -4,7 +4,7 @@ from six.moves.urllib.parse import urlparse
from
scrapy.http
import
Response
,
Request
from
scrapy.spider
import
Spider
from
scrapy.
contrib.spidermiddleware
.offsite
import
OffsiteMiddleware
from
scrapy.
spidermiddlewares
.offsite
import
OffsiteMiddleware
from
scrapy.utils.test
import
get_crawler
class
TestOffsiteMiddleware
(
TestCase
):
...
...
tests/test_spidermiddleware_referer.py
浏览文件 @
180272c0
...
...
@@ -2,7 +2,7 @@ from unittest import TestCase
from
scrapy.http
import
Response
,
Request
from
scrapy.spider
import
Spider
from
scrapy.
contrib.spidermiddleware
.referer
import
RefererMiddleware
from
scrapy.
spidermiddlewares
.referer
import
RefererMiddleware
class
TestRefererMiddleware
(
TestCase
):
...
...
tests/test_spidermiddleware_urllength.py
浏览文件 @
180272c0
from
unittest
import
TestCase
from
scrapy.
contrib.spidermiddleware
.urllength
import
UrlLengthMiddleware
from
scrapy.
spidermiddlewares
.urllength
import
UrlLengthMiddleware
from
scrapy.http
import
Response
,
Request
from
scrapy.spider
import
Spider
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录