Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
looyolo
scrapy
提交
9a3e3ba5
S
scrapy
项目概览
looyolo
/
scrapy
与 Fork 源项目一致
从无法访问的项目Fork
通知
2
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
S
scrapy
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
9a3e3ba5
编写于
4月 21, 2015
作者:
J
Julia Medina
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Move scrapy/contrib remaining top-level files to scrapy/extensions
上级
e262c5b8
变更
23
隐藏空白更改
内联
并排
Showing
23 changed file
with
69 addition
and
80 deletion
+69
-80
docs/topics/downloader-middleware.rst
docs/topics/downloader-middleware.rst
+7
-7
docs/topics/extensions.rst
docs/topics/extensions.rst
+15
-15
docs/topics/feed-exports.rst
docs/topics/feed-exports.rst
+6
-6
docs/topics/settings.rst
docs/topics/settings.rst
+14
-14
scrapy/downloadermiddlewares/httpcache.py
scrapy/downloadermiddlewares/httpcache.py
+0
-13
scrapy/extensions/__init__.py
scrapy/extensions/__init__.py
+0
-0
scrapy/extensions/closespider.py
scrapy/extensions/closespider.py
+0
-0
scrapy/extensions/corestats.py
scrapy/extensions/corestats.py
+0
-0
scrapy/extensions/debug.py
scrapy/extensions/debug.py
+0
-0
scrapy/extensions/feedexport.py
scrapy/extensions/feedexport.py
+0
-0
scrapy/extensions/httpcache.py
scrapy/extensions/httpcache.py
+0
-0
scrapy/extensions/logstats.py
scrapy/extensions/logstats.py
+0
-0
scrapy/extensions/memdebug.py
scrapy/extensions/memdebug.py
+0
-0
scrapy/extensions/memusage.py
scrapy/extensions/memusage.py
+0
-0
scrapy/extensions/spiderstate.py
scrapy/extensions/spiderstate.py
+0
-0
scrapy/extensions/statsmailer.py
scrapy/extensions/statsmailer.py
+0
-0
scrapy/extensions/throttle.py
scrapy/extensions/throttle.py
+0
-0
scrapy/settings/default_settings.py
scrapy/settings/default_settings.py
+15
-15
scrapy/templates/project/module/settings.py.tmpl
scrapy/templates/project/module/settings.py.tmpl
+1
-1
tests/py3-ignores.txt
tests/py3-ignores.txt
+2
-0
tests/test_downloadermiddleware_httpcache.py
tests/test_downloadermiddleware_httpcache.py
+7
-7
tests/test_feedexport.py
tests/test_feedexport.py
+1
-1
tests/test_spiderstate.py
tests/test_spiderstate.py
+1
-1
未找到文件。
docs/topics/downloader-middleware.rst
浏览文件 @
9a3e3ba5
...
...
@@ -349,7 +349,7 @@ when an Internet connection is not available. The goal is to be able to
In order to use this policy, set:
* :setting:`HTTPCACHE_POLICY` to ``scrapy.
contrib
.httpcache.DummyPolicy``
* :setting:`HTTPCACHE_POLICY` to ``scrapy.
extensions
.httpcache.DummyPolicy``
.. _httpcache-policy-rfc2616:
...
...
@@ -383,7 +383,7 @@ what is missing:
In order to use this policy, set:
* :setting:`HTTPCACHE_POLICY` to ``scrapy.
contrib
.httpcache.RFC2616Policy``
* :setting:`HTTPCACHE_POLICY` to ``scrapy.
extensions
.httpcache.RFC2616Policy``
.. _httpcache-storage-fs:
...
...
@@ -395,7 +395,7 @@ File system storage backend is available for the HTTP cache middleware.
In order to use this storage backend, set:
* :setting:`HTTPCACHE_STORAGE` to ``scrapy.
contrib
.httpcache.FilesystemCacheStorage``
* :setting:`HTTPCACHE_STORAGE` to ``scrapy.
extensions
.httpcache.FilesystemCacheStorage``
Each request/response pair is stored in a different directory containing
the following files:
...
...
@@ -430,7 +430,7 @@ By default, it uses the anydbm_ module, but you can change it with the
In order to use this storage backend, set:
* :setting:`HTTPCACHE_STORAGE` to ``scrapy.
contrib
.httpcache.DbmCacheStorage``
* :setting:`HTTPCACHE_STORAGE` to ``scrapy.
extensions
.httpcache.DbmCacheStorage``
.. _httpcache-storage-leveldb:
...
...
@@ -447,7 +447,7 @@ the scrapy shell in parallel for the same spider.
In order to use this storage backend:
* set :setting:`HTTPCACHE_STORAGE` to ``scrapy.
contrib
.httpcache.LeveldbCacheStorage``
* set :setting:`HTTPCACHE_STORAGE` to ``scrapy.
extensions
.httpcache.LeveldbCacheStorage``
* install `LevelDB python bindings`_ like ``pip install leveldb``
.. _LevelDB: http://code.google.com/p/leveldb/
...
...
@@ -536,7 +536,7 @@ Don't cache responses with these URI schemes.
HTTPCACHE_STORAGE
^^^^^^^^^^^^^^^^^
Default: ``'scrapy.
contrib
.httpcache.FilesystemCacheStorage'``
Default: ``'scrapy.
extensions
.httpcache.FilesystemCacheStorage'``
The class which implements the cache storage backend.
...
...
@@ -559,7 +559,7 @@ HTTPCACHE_POLICY
.. versionadded:: 0.18
Default: ``'scrapy.
contrib
.httpcache.DummyPolicy'``
Default: ``'scrapy.
extensions
.httpcache.DummyPolicy'``
The class which implements the cache policy.
...
...
docs/topics/extensions.rst
浏览文件 @
9a3e3ba5
...
...
@@ -35,7 +35,7 @@ your Scrapy settings. In :setting:`EXTENSIONS`, each extension is represented
by a string: the full Python path to the extension's class name. For example::
EXTENSIONS = {
'scrapy.
contrib
.corestats.CoreStats': 500,
'scrapy.
extensions
.corestats.CoreStats': 500,
'scrapy.telnet.TelnetConsole': 500,
}
...
...
@@ -69,7 +69,7 @@ included in the :setting:`EXTENSIONS_BASE` setting) you must set its order to
``None``. For example::
EXTENSIONS = {
'scrapy.
contrib
.corestats.CoreStats': None,
'scrapy.
extensions
.corestats.CoreStats': None,
}
Writing your own extension
...
...
@@ -158,7 +158,7 @@ General purpose extensions
Log Stats extension
~~~~~~~~~~~~~~~~~~~
.. module:: scrapy.
contrib
.logstats
.. module:: scrapy.
extensions
.logstats
:synopsis: Basic stats logging
.. class:: LogStats
...
...
@@ -168,7 +168,7 @@ Log basic stats like crawled pages and scraped items.
Core Stats extension
~~~~~~~~~~~~~~~~~~~~
.. module:: scrapy.
contrib
.corestats
.. module:: scrapy.
extensions
.corestats
:synopsis: Core stats collection
.. class:: CoreStats
...
...
@@ -198,10 +198,10 @@ setting, and the server will listen in the port specified in
Memory usage extension
~~~~~~~~~~~~~~~~~~~~~~
.. module:: scrapy.
contrib
.memusage
.. module:: scrapy.
extensions
.memusage
:synopsis: Memory usage extension
.. class:: scrapy.
contrib
.memusage.MemoryUsage
.. class:: scrapy.
extensions
.memusage.MemoryUsage
.. note:: This extension does not work in Windows.
...
...
@@ -226,10 +226,10 @@ can be configured with the following settings:
Memory debugger extension
~~~~~~~~~~~~~~~~~~~~~~~~~
.. module:: scrapy.
contrib
.memdebug
.. module:: scrapy.
extensions
.memdebug
:synopsis: Memory debugger extension
.. class:: scrapy.
contrib
.memdebug.MemoryDebugger
.. class:: scrapy.
extensions
.memdebug.MemoryDebugger
An extension for debugging memory usage. It collects information about:
...
...
@@ -242,10 +242,10 @@ info will be stored in the stats.
Close spider extension
~~~~~~~~~~~~~~~~~~~~~~
.. module:: scrapy.
contrib
.closespider
.. module:: scrapy.
extensions
.closespider
:synopsis: Close spider extension
.. class:: scrapy.
contrib
.closespider.CloseSpider
.. class:: scrapy.
extensions
.closespider.CloseSpider
Closes a spider automatically when some conditions are met, using a specific
closing reason for each condition.
...
...
@@ -313,17 +313,17 @@ set), spiders won't be closed by number of errors.
StatsMailer extension
~~~~~~~~~~~~~~~~~~~~~
.. module:: scrapy.
contrib
.statsmailer
.. module:: scrapy.
extensions
.statsmailer
:synopsis: StatsMailer extension
.. class:: scrapy.
contrib
.statsmailer.StatsMailer
.. class:: scrapy.
extensions
.statsmailer.StatsMailer
This simple extension can be used to send a notification e-mail every time a
domain has finished scraping, including the Scrapy stats collected. The email
will be sent to all recipients specified in the :setting:`STATSMAILER_RCPTS`
setting.
.. module:: scrapy.
contrib
.debug
.. module:: scrapy.
extensions
.debug
:synopsis: Extensions for debugging Scrapy
Debugging extensions
...
...
@@ -332,7 +332,7 @@ Debugging extensions
Stack trace dump extension
~~~~~~~~~~~~~~~~~~~~~~~~~~
.. class:: scrapy.
contrib
.debug.StackTraceDump
.. class:: scrapy.
extensions
.debug.StackTraceDump
Dumps information about the running process when a `SIGQUIT`_ or `SIGUSR2`_
signal is received. The information dumped is the following:
...
...
@@ -361,7 +361,7 @@ There are at least two ways to send Scrapy the `SIGQUIT`_ signal:
Debugger extension
~~~~~~~~~~~~~~~~~~
.. class:: scrapy.
contrib
.debug.Debugger
.. class:: scrapy.
extensions
.debug.Debugger
Invokes a `Python debugger`_ inside a running Scrapy process when a `SIGUSR2`_
signal is received. After the debugger is exited, the Scrapy process continues
...
...
docs/topics/feed-exports.rst
浏览文件 @
9a3e3ba5
...
...
@@ -209,7 +209,7 @@ These are the settings used for configuring the feed exports:
* :setting:`FEED_STORE_EMPTY`
* :setting:`FEED_EXPORT_FIELDS`
.. currentmodule:: scrapy.
contrib
.feedexport
.. currentmodule:: scrapy.
extensions
.feedexport
.. setting:: FEED_URI
...
...
@@ -272,11 +272,11 @@ FEED_STORAGES_BASE
Default::
{
'': 'scrapy.
contrib
.feedexport.FileFeedStorage',
'file': 'scrapy.
contrib
.feedexport.FileFeedStorage',
'stdout': 'scrapy.
contrib
.feedexport.StdoutFeedStorage',
's3': 'scrapy.
contrib
.feedexport.S3FeedStorage',
'ftp': 'scrapy.
contrib
.feedexport.FTPFeedStorage',
'': 'scrapy.
extensions
.feedexport.FileFeedStorage',
'file': 'scrapy.
extensions
.feedexport.FileFeedStorage',
'stdout': 'scrapy.
extensions
.feedexport.StdoutFeedStorage',
's3': 'scrapy.
extensions
.feedexport.S3FeedStorage',
'ftp': 'scrapy.
extensions
.feedexport.FTPFeedStorage',
}
A dict containing the built-in feed storage backends supported by Scrapy.
...
...
docs/topics/settings.rst
浏览文件 @
9a3e3ba5
...
...
@@ -536,15 +536,15 @@ EXTENSIONS_BASE
Default::
{
'scrapy.
contrib
.corestats.CoreStats': 0,
'scrapy.
extensions
.corestats.CoreStats': 0,
'scrapy.telnet.TelnetConsole': 0,
'scrapy.
contrib
.memusage.MemoryUsage': 0,
'scrapy.
contrib
.memdebug.MemoryDebugger': 0,
'scrapy.
contrib
.closespider.CloseSpider': 0,
'scrapy.
contrib
.feedexport.FeedExporter': 0,
'scrapy.
contrib
.logstats.LogStats': 0,
'scrapy.
contrib
.spiderstate.SpiderState': 0,
'scrapy.
contrib
.throttle.AutoThrottle': 0,
'scrapy.
extensions
.memusage.MemoryUsage': 0,
'scrapy.
extensions
.memdebug.MemoryDebugger': 0,
'scrapy.
extensions
.closespider.CloseSpider': 0,
'scrapy.
extensions
.feedexport.FeedExporter': 0,
'scrapy.
extensions
.logstats.LogStats': 0,
'scrapy.
extensions
.spiderstate.SpiderState': 0,
'scrapy.
extensions
.throttle.AutoThrottle': 0,
}
The list of available extensions. Keep in mind that some of them need to
...
...
@@ -689,7 +689,7 @@ MEMUSAGE_ENABLED
Default: ``False``
Scope: ``scrapy.
contrib
.memusage``
Scope: ``scrapy.
extensions
.memusage``
Whether to enable the memory usage extension that will shutdown the Scrapy
process when it exceeds a memory limit, and also notify by email when that
...
...
@@ -704,7 +704,7 @@ MEMUSAGE_LIMIT_MB
Default: ``0``
Scope: ``scrapy.
contrib
.memusage``
Scope: ``scrapy.
extensions
.memusage``
The maximum amount of memory to allow (in megabytes) before shutting down
Scrapy (if MEMUSAGE_ENABLED is True). If zero, no check will be performed.
...
...
@@ -718,7 +718,7 @@ MEMUSAGE_NOTIFY_MAIL
Default: ``False``
Scope: ``scrapy.
contrib
.memusage``
Scope: ``scrapy.
extensions
.memusage``
A list of emails to notify if the memory limit has been reached.
...
...
@@ -735,7 +735,7 @@ MEMUSAGE_REPORT
Default: ``False``
Scope: ``scrapy.
contrib
.memusage``
Scope: ``scrapy.
extensions
.memusage``
Whether to send a memory usage report after each spider has been closed.
...
...
@@ -748,7 +748,7 @@ MEMUSAGE_WARNING_MB
Default: ``0``
Scope: ``scrapy.
contrib
.memusage``
Scope: ``scrapy.
extensions
.memusage``
The maximum amount of memory to allow (in megabytes) before sending a warning
email notifying about it. If zero, no warning will be produced.
...
...
@@ -961,7 +961,7 @@ STATSMAILER_RCPTS
Default: ``[]`` (empty list)
Send Scrapy stats after spiders finish scraping. See
:class:`~scrapy.
contrib
.statsmailer.StatsMailer` for more info.
:class:`~scrapy.
extensions
.statsmailer.StatsMailer` for more info.
.. setting:: TELNETCONSOLE_ENABLED
...
...
scrapy/downloadermiddlewares/httpcache.py
浏览文件 @
9a3e3ba5
...
...
@@ -90,16 +90,3 @@ class HttpCacheMiddleware(object):
self
.
storage
.
store_response
(
spider
,
request
,
response
)
else
:
self
.
stats
.
inc_value
(
'httpcache/uncacheable'
,
spider
=
spider
)
from
scrapy.contrib.httpcache
import
FilesystemCacheStorage
as
_FilesystemCacheStorage
class
FilesystemCacheStorage
(
_FilesystemCacheStorage
):
def
__init__
(
self
,
*
args
,
**
kwargs
):
import
warnings
from
scrapy.exceptions
import
ScrapyDeprecationWarning
warnings
.
warn
(
'Importing FilesystemCacheStorage from '
'scrapy.contrib.downloadermiddlware.httpcache is '
'deprecated, use scrapy.contrib.httpcache instead.'
,
category
=
ScrapyDeprecationWarning
,
stacklevel
=
1
)
super
(
FilesystemCacheStorage
,
self
).
__init__
(
*
args
,
**
kwargs
)
scrapy/extensions/__init__.py
0 → 100644
浏览文件 @
9a3e3ba5
scrapy/
contrib
/closespider.py
→
scrapy/
extensions
/closespider.py
浏览文件 @
9a3e3ba5
文件已移动
scrapy/
contrib
/corestats.py
→
scrapy/
extensions
/corestats.py
浏览文件 @
9a3e3ba5
文件已移动
scrapy/
contrib
/debug.py
→
scrapy/
extensions
/debug.py
浏览文件 @
9a3e3ba5
文件已移动
scrapy/
contrib
/feedexport.py
→
scrapy/
extensions
/feedexport.py
浏览文件 @
9a3e3ba5
文件已移动
scrapy/
contrib
/httpcache.py
→
scrapy/
extensions
/httpcache.py
浏览文件 @
9a3e3ba5
文件已移动
scrapy/
contrib
/logstats.py
→
scrapy/
extensions
/logstats.py
浏览文件 @
9a3e3ba5
文件已移动
scrapy/
contrib
/memdebug.py
→
scrapy/
extensions
/memdebug.py
浏览文件 @
9a3e3ba5
文件已移动
scrapy/
contrib
/memusage.py
→
scrapy/
extensions
/memusage.py
浏览文件 @
9a3e3ba5
文件已移动
scrapy/
contrib
/spiderstate.py
→
scrapy/
extensions
/spiderstate.py
浏览文件 @
9a3e3ba5
文件已移动
scrapy/
contrib
/statsmailer.py
→
scrapy/
extensions
/statsmailer.py
浏览文件 @
9a3e3ba5
文件已移动
scrapy/
contrib
/throttle.py
→
scrapy/
extensions
/throttle.py
浏览文件 @
9a3e3ba5
文件已移动
scrapy/settings/default_settings.py
浏览文件 @
9a3e3ba5
...
...
@@ -113,15 +113,15 @@ except KeyError:
EXTENSIONS
=
{}
EXTENSIONS_BASE
=
{
'scrapy.
contrib
.corestats.CoreStats'
:
0
,
'scrapy.
extensions
.corestats.CoreStats'
:
0
,
'scrapy.telnet.TelnetConsole'
:
0
,
'scrapy.
contrib
.memusage.MemoryUsage'
:
0
,
'scrapy.
contrib
.memdebug.MemoryDebugger'
:
0
,
'scrapy.
contrib
.closespider.CloseSpider'
:
0
,
'scrapy.
contrib
.feedexport.FeedExporter'
:
0
,
'scrapy.
contrib
.logstats.LogStats'
:
0
,
'scrapy.
contrib
.spiderstate.SpiderState'
:
0
,
'scrapy.
contrib
.throttle.AutoThrottle'
:
0
,
'scrapy.
extensions
.memusage.MemoryUsage'
:
0
,
'scrapy.
extensions
.memdebug.MemoryDebugger'
:
0
,
'scrapy.
extensions
.closespider.CloseSpider'
:
0
,
'scrapy.
extensions
.feedexport.FeedExporter'
:
0
,
'scrapy.
extensions
.logstats.LogStats'
:
0
,
'scrapy.
extensions
.spiderstate.SpiderState'
:
0
,
'scrapy.
extensions
.throttle.AutoThrottle'
:
0
,
}
FEED_URI
=
None
...
...
@@ -131,11 +131,11 @@ FEED_STORE_EMPTY = False
FEED_EXPORT_FIELDS
=
None
FEED_STORAGES
=
{}
FEED_STORAGES_BASE
=
{
''
:
'scrapy.
contrib
.feedexport.FileFeedStorage'
,
'file'
:
'scrapy.
contrib
.feedexport.FileFeedStorage'
,
'stdout'
:
'scrapy.
contrib
.feedexport.StdoutFeedStorage'
,
's3'
:
'scrapy.
contrib
.feedexport.S3FeedStorage'
,
'ftp'
:
'scrapy.
contrib
.feedexport.FTPFeedStorage'
,
''
:
'scrapy.
extensions
.feedexport.FileFeedStorage'
,
'file'
:
'scrapy.
extensions
.feedexport.FileFeedStorage'
,
'stdout'
:
'scrapy.
extensions
.feedexport.StdoutFeedStorage'
,
's3'
:
'scrapy.
extensions
.feedexport.S3FeedStorage'
,
'ftp'
:
'scrapy.
extensions
.feedexport.FTPFeedStorage'
,
}
FEED_EXPORTERS
=
{}
FEED_EXPORTERS_BASE
=
{
...
...
@@ -151,12 +151,12 @@ FEED_EXPORTERS_BASE = {
HTTPCACHE_ENABLED
=
False
HTTPCACHE_DIR
=
'httpcache'
HTTPCACHE_IGNORE_MISSING
=
False
HTTPCACHE_STORAGE
=
'scrapy.
contrib
.httpcache.FilesystemCacheStorage'
HTTPCACHE_STORAGE
=
'scrapy.
extensions
.httpcache.FilesystemCacheStorage'
HTTPCACHE_EXPIRATION_SECS
=
0
HTTPCACHE_IGNORE_HTTP_CODES
=
[]
HTTPCACHE_IGNORE_SCHEMES
=
[
'file'
]
HTTPCACHE_DBM_MODULE
=
'anydbm'
HTTPCACHE_POLICY
=
'scrapy.
contrib
.httpcache.DummyPolicy'
HTTPCACHE_POLICY
=
'scrapy.
extensions
.httpcache.DummyPolicy'
HTTPCACHE_GZIP
=
False
ITEM_PROCESSOR
=
'scrapy.pipelines.ItemPipelineManager'
...
...
scrapy/templates/project/module/settings.py.tmpl
浏览文件 @
9a3e3ba5
...
...
@@ -82,4 +82,4 @@ NEWSPIDER_MODULE = '$project_name.spiders'
#HTTPCACHE_EXPIRATION_SECS=0
#HTTPCACHE_DIR='httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES=[]
#HTTPCACHE_STORAGE='scrapy.
contrib
.httpcache.FilesystemCacheStorage'
#HTTPCACHE_STORAGE='scrapy.
extensions
.httpcache.FilesystemCacheStorage'
tests/py3-ignores.txt
浏览文件 @
9a3e3ba5
...
...
@@ -100,7 +100,9 @@ scrapy/downloadermiddlewares/retry.py
scrapy/downloadermiddlewares/httpproxy.py
scrapy/downloadermiddlewares/cookies.py
scrapy/contrib/statsmailer.py
scrapy/extensions/statsmailer.py
scrapy/contrib/memusage.py
scrapy/extensions/memusage.py
scrapy/commands/deploy.py
scrapy/commands/bench.py
scrapy/mail.py
tests/test_downloadermiddleware_httpcache.py
浏览文件 @
9a3e3ba5
...
...
@@ -17,8 +17,8 @@ from scrapy.downloadermiddlewares.httpcache import HttpCacheMiddleware
class
_BaseTest
(
unittest
.
TestCase
):
storage_class
=
'scrapy.
contrib
.httpcache.DbmCacheStorage'
policy_class
=
'scrapy.
contrib
.httpcache.RFC2616Policy'
storage_class
=
'scrapy.
extensions
.httpcache.DbmCacheStorage'
policy_class
=
'scrapy.
extensions
.httpcache.RFC2616Policy'
def
setUp
(
self
):
self
.
yesterday
=
email
.
utils
.
formatdate
(
time
.
time
()
-
86400
)
...
...
@@ -127,7 +127,7 @@ class DefaultStorageTest(_BaseTest):
class
DbmStorageTest
(
DefaultStorageTest
):
storage_class
=
'scrapy.
contrib
.httpcache.DbmCacheStorage'
storage_class
=
'scrapy.
extensions
.httpcache.DbmCacheStorage'
class
DbmStorageWithCustomDbmModuleTest
(
DbmStorageTest
):
...
...
@@ -146,7 +146,7 @@ class DbmStorageWithCustomDbmModuleTest(DbmStorageTest):
class
FilesystemStorageTest
(
DefaultStorageTest
):
storage_class
=
'scrapy.
contrib
.httpcache.FilesystemCacheStorage'
storage_class
=
'scrapy.
extensions
.httpcache.FilesystemCacheStorage'
class
FilesystemStorageGzipTest
(
FilesystemStorageTest
):
...
...
@@ -157,12 +157,12 @@ class FilesystemStorageGzipTest(FilesystemStorageTest):
class
LeveldbStorageTest
(
DefaultStorageTest
):
pytest
.
importorskip
(
'leveldb'
)
storage_class
=
'scrapy.
contrib
.httpcache.LeveldbCacheStorage'
storage_class
=
'scrapy.
extensions
.httpcache.LeveldbCacheStorage'
class
DummyPolicyTest
(
_BaseTest
):
policy_class
=
'scrapy.
contrib
.httpcache.DummyPolicy'
policy_class
=
'scrapy.
extensions
.httpcache.DummyPolicy'
def
test_middleware
(
self
):
with
self
.
_middleware
()
as
mw
:
...
...
@@ -254,7 +254,7 @@ class DummyPolicyTest(_BaseTest):
class
RFC2616PolicyTest
(
DefaultStorageTest
):
policy_class
=
'scrapy.
contrib
.httpcache.RFC2616Policy'
policy_class
=
'scrapy.
extensions
.httpcache.RFC2616Policy'
def
_process_requestresponse
(
self
,
mw
,
request
,
response
):
try
:
...
...
tests/test_
contrib_
feedexport.py
→
tests/test_feedexport.py
浏览文件 @
9a3e3ba5
...
...
@@ -16,7 +16,7 @@ from tests.mockserver import MockServer
from
w3lib.url
import
path_to_file_uri
import
scrapy
from
scrapy.
contrib
.feedexport
import
(
from
scrapy.
extensions
.feedexport
import
(
IFeedStorage
,
FileFeedStorage
,
FTPFeedStorage
,
S3FeedStorage
,
StdoutFeedStorage
)
...
...
tests/test_
contrib_
spiderstate.py
→
tests/test_spiderstate.py
浏览文件 @
9a3e3ba5
...
...
@@ -2,7 +2,7 @@ import os
from
datetime
import
datetime
from
twisted.trial
import
unittest
from
scrapy.
contrib
.spiderstate
import
SpiderState
from
scrapy.
extensions
.spiderstate
import
SpiderState
from
scrapy.spider
import
Spider
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录