Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
looyolo
scrapy
提交
7e258716
S
scrapy
项目概览
looyolo
/
scrapy
与 Fork 源项目一致
从无法访问的项目Fork
通知
2
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
S
scrapy
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
7e258716
编写于
9月 04, 2009
作者:
I
Ismael Carnales
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
added missing middleware docs
上级
6d127d7f
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
114 addition
and
9 deletion
+114
-9
docs/topics/downloader-middleware.rst
docs/topics/downloader-middleware.rst
+104
-9
docs/topics/spider-middleware.rst
docs/topics/spider-middleware.rst
+10
-0
未找到文件。
docs/topics/downloader-middleware.rst
浏览文件 @
7e258716
...
...
@@ -142,6 +142,16 @@ middleware, see the :ref:`downloader middleware usage guide
For a list of the components enabled by default (and their orders) see the
:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting.
CookiesMiddleware
-----------------
.. module:: scrapy.contrib.downloadermiddleware.cookies
:synopsis: Cookies Downloader Middleware
.. class:: CookiesMiddleware
This middleware enables working with sites that need cookies.
DefaultHeadersMiddleware
------------------------
...
...
@@ -199,24 +209,81 @@ HttpCacheMiddleware
The :class:`HttpCacheMiddleware` can be configured through the following
settings (see the settings documentation for more info):
* :setting:`HTTPCACHE_DIR` - this one actually enables the cache besides
settings the cache dir
* :setting:`HTTPCACHE_IGNORE_MISSING` - ignoring missing requests instead
of downloading them
* :setting:`HTTPCACHE_SECTORIZE` - split HTTP cache in several directories
(for performance reasons)
* :setting:`HTTPCACHE_EXPIRATION_SECS` - how many secs until the cache is
considered out of date
* :setting:`HTTPCACHE_DIR` - this one actually enables the cache besides
settings the cache dir
* :setting:`HTTPCACHE_IGNORE_MISSING` - ignoring missing requests instead
of downloading them
* :setting:`HTTPCACHE_SECTORIZE` - split HTTP cache in several directories
(for performance reasons)
* :setting:`HTTPCACHE_EXPIRATION_SECS` - how many secs until the cache is
considered out of date
.. _topics-dlmw-robots:
HttpCompressionMiddleware
-------------------------
.. module:: scrapy.contrib.downloadermiddleware.httpcompression
:synopsis: Http Compression Middleware
.. class:: HttpCompressionMiddleware
This middleware allows compressed (gzip, deflate) traffic to be
sent/received from web sites.
RedirectMiddleware
-------------------
.. module:: scrapy.contrib.downloadermiddleware.redirect
:synopsis: Redirection Middleware
.. class:: RedirectMiddleware
This middlware handles redirection of requests based on response status and
meta-refresh html tag.
The :class:`RedirectMiddleware` can be configured through the following
settings (see the settings documentation for more info):
* :setting:`REDIRECT_MAX_METAREFRESH_DELAY` - Maximum meta-refresh delay that a page is allowed to have for redirection.
* :setting:`REDIRECT_MAX_TIMES` - Maximum number of redirects to perform on a request.
* :setting:`REDIRECT_PRIORITY_ADJUST` - Adjusts the redirected request priority by this amount.
RetryMiddleware
---------------
.. module:: scrapy.contrib.downloadermiddleware.retry
:synopsis: Retry Middleware
.. class:: RetryMiddleware
A middlware to retry failed requests that are potentially caused by
temporary problems such as a connection timeout or HTTP 500 error.
Failed pages are collected on the scraping process and rescheduled at the
end, once the spider has finished crawling all regular (non failed) pages.
Once there is no more failed pages to retry this middleware sends a signal
(retry_complete), so other extensions could connect to that signal.
The :class:`RetryMiddleware` can be configured through the following
settings (see the settings documentation for more info):
* :setting:`RETRY_TIMES` - how many times to retry a failed page
* :setting:`RETRY_HTTP_CODES` - which HTTP response codes to retry
About HTTP errors to consider:
You may want to remove 400 from RETRY_HTTP_CODES, if you stick to the
HTTP protocol. It's included by default because it's a common code used
to indicate server overload, which would be something we want to retry.
RobotsTxtMiddleware
-------------------
.. module:: scrapy.contrib.downloadermiddleware.robotstxt
:synopsis: robots.txt middleware
.. class:: RobotsTxtMiddleware
:
.. class:: RobotsTxtMiddleware
This middleware filters out requests forbidden by the robots.txt exclusion
standard.
...
...
@@ -229,3 +296,31 @@ RobotsTxtMiddleware
if they were requested before the robots.txt file was downloaded. This
is a known limitation of the current robots.txt middleware and will
be fixed in the future.
DownloaderStats
---------------
.. module:: scrapy.contrib.downloadermiddleware.stats
:synopsis: Downloader Stats Middleware
.. class:: DownloaderStats
Middleware that store stats of all requests, responses and exceptions that
pass through it.
To use this middleware you must enable the :setting:`DOWNLOADER_STATS`
setting.
UserAgentMiddleware
-------------------
.. module:: scrapy.contrib.downloadermiddleware.useragent
:synopsis: User Agent Middleware
.. class:: UserAgentMiddleware
Middleware that allows spiders to override the default user agent.
In order for a spider to override the default user agent, its `user_agent`
attribute must be set.
docs/topics/spider-middleware.rst
浏览文件 @
7e258716
...
...
@@ -214,6 +214,16 @@ OffsiteMiddleware
Spider can add more domains to exclude using
:attr:`~scrapy.spider.BaseSpider.extra_domain_names` attribute.
RefererMiddleware
-----------------
.. module:: scrapy.contrib.spidermiddleware.referer
:synopsis: Referer Spider Middleware
.. class:: RefererMiddleware
Populates Request referer field, based on the Response which originated it.
RequestLimitMiddleware
----------------------
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录