Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
looyolo
scrapy
提交
108f8c4f
S
scrapy
项目概览
looyolo
/
scrapy
与 Fork 源项目一致
从无法访问的项目Fork
通知
2
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
S
scrapy
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
108f8c4f
编写于
10月 28, 2017
作者:
M
Mikhail Korobov
提交者:
GitHub
10月 28, 2017
浏览文件
操作
浏览文件
下载
差异文件
Merge pull request #2982 from codeaditya/https-links
Fix broken links and use https links wherever possible
上级
79df51aa
dae7b1cd
变更
39
隐藏空白更改
内联
并排
Showing
39 changed file
with
80 addition
and
80 deletion
+80
-80
CONTRIBUTING.md
CONTRIBUTING.md
+2
-2
INSTALL
INSTALL
+1
-1
README.rst
README.rst
+7
-7
debian/control
debian/control
+3
-3
debian/copyright
debian/copyright
+4
-4
docs/contributing.rst
docs/contributing.rst
+1
-1
docs/intro/overview.rst
docs/intro/overview.rst
+1
-1
docs/topics/practices.rst
docs/topics/practices.rst
+1
-1
docs/topics/selectors.rst
docs/topics/selectors.rst
+2
-2
docs/topics/shell.rst
docs/topics/shell.rst
+4
-4
scrapy/_monkeypatches.py
scrapy/_monkeypatches.py
+2
-2
scrapy/core/downloader/contextfactory.py
scrapy/core/downloader/contextfactory.py
+2
-2
scrapy/crawler.py
scrapy/crawler.py
+1
-1
scrapy/downloadermiddlewares/chunked.py
scrapy/downloadermiddlewares/chunked.py
+1
-1
scrapy/downloadermiddlewares/httpcache.py
scrapy/downloadermiddlewares/httpcache.py
+1
-1
scrapy/exporters.py
scrapy/exporters.py
+1
-1
scrapy/extensions/httpcache.py
scrapy/extensions/httpcache.py
+5
-5
scrapy/extensions/telnet.py
scrapy/extensions/telnet.py
+1
-1
scrapy/pipelines/files.py
scrapy/pipelines/files.py
+2
-2
scrapy/signalmanager.py
scrapy/signalmanager.py
+1
-1
scrapy/templates/project/module/items.py.tmpl
scrapy/templates/project/module/items.py.tmpl
+1
-1
scrapy/templates/project/module/middlewares.py.tmpl
scrapy/templates/project/module/middlewares.py.tmpl
+1
-1
scrapy/templates/project/module/pipelines.py.tmpl
scrapy/templates/project/module/pipelines.py.tmpl
+1
-1
scrapy/templates/project/module/settings.py.tmpl
scrapy/templates/project/module/settings.py.tmpl
+10
-10
scrapy/templates/project/scrapy.cfg
scrapy/templates/project/scrapy.cfg
+1
-1
scrapy/utils/defer.py
scrapy/utils/defer.py
+1
-1
scrapy/utils/deprecate.py
scrapy/utils/deprecate.py
+6
-6
scrapy/utils/http.py
scrapy/utils/http.py
+1
-1
scrapy/utils/log.py
scrapy/utils/log.py
+1
-1
scrapy/utils/url.py
scrapy/utils/url.py
+1
-1
sep/sep-001.rst
sep/sep-001.rst
+1
-1
sep/sep-006.rst
sep/sep-006.rst
+4
-4
sep/sep-013.rst
sep/sep-013.rst
+1
-1
sep/sep-017.rst
sep/sep-017.rst
+1
-1
sep/sep-020.rst
sep/sep-020.rst
+1
-1
setup.py
setup.py
+1
-1
tests/__init__.py
tests/__init__.py
+1
-1
tests/keys/example-com.conf
tests/keys/example-com.conf
+2
-2
tox.ini
tox.ini
+1
-1
未找到文件。
CONTRIBUTING.md
浏览文件 @
108f8c4f
The guidelines for contributing are available here:
The guidelines for contributing are available here:
http://doc.scrapy.org/en/master/contributing.html
http
s
://doc.scrapy.org/en/master/contributing.html
Please do not abuse the issue tracker for support questions.
Please do not abuse the issue tracker for support questions.
If your issue topic can be rephrased to "How to ...?", please use the
If your issue topic can be rephrased to "How to ...?", please use the
support channels to get it answered: http://scrapy.org/community/
support channels to get it answered: http
s
://scrapy.org/community/
INSTALL
浏览文件 @
108f8c4f
For information about installing Scrapy see:
For information about installing Scrapy see:
* docs/intro/install.rst (local file)
* docs/intro/install.rst (local file)
* http://doc.scrapy.org/en/latest/intro/install.html (online version)
* http
s
://doc.scrapy.org/en/latest/intro/install.html (online version)
README.rst
浏览文件 @
108f8c4f
...
@@ -31,7 +31,7 @@ crawl websites and extract structured data from their pages. It can be used for
...
@@ -31,7 +31,7 @@ crawl websites and extract structured data from their pages. It can be used for
a wide range of purposes, from data mining to monitoring and automated testing.
a wide range of purposes, from data mining to monitoring and automated testing.
For more information including a list of features check the Scrapy homepage at:
For more information including a list of features check the Scrapy homepage at:
http://scrapy.org
http
s
://scrapy.org
Requirements
Requirements
============
============
...
@@ -47,12 +47,12 @@ The quick way::
...
@@ -47,12 +47,12 @@ The quick way::
pip install scrapy
pip install scrapy
For more details see the install section in the documentation:
For more details see the install section in the documentation:
http://doc.scrapy.org/en/latest/intro/install.html
http
s
://doc.scrapy.org/en/latest/intro/install.html
Documentation
Documentation
=============
=============
Documentation is available online at http://doc.scrapy.org/ and in the ``docs``
Documentation is available online at http
s
://doc.scrapy.org/ and in the ``docs``
directory.
directory.
Releases
Releases
...
@@ -63,12 +63,12 @@ You can find release notes at https://doc.scrapy.org/en/latest/news.html
...
@@ -63,12 +63,12 @@ You can find release notes at https://doc.scrapy.org/en/latest/news.html
Community (blog, twitter, mail list, IRC)
Community (blog, twitter, mail list, IRC)
=========================================
=========================================
See http://scrapy.org/community/
See http
s
://scrapy.org/community/
Contributing
Contributing
============
============
See http://doc.scrapy.org/en/master/contributing.html
See http
s
://doc.scrapy.org/en/master/contributing.html
Code of Conduct
Code of Conduct
---------------
---------------
...
@@ -82,9 +82,9 @@ Please report unacceptable behavior to opensource@scrapinghub.com.
...
@@ -82,9 +82,9 @@ Please report unacceptable behavior to opensource@scrapinghub.com.
Companies using Scrapy
Companies using Scrapy
======================
======================
See http://scrapy.org/companies/
See http
s
://scrapy.org/companies/
Commercial Support
Commercial Support
==================
==================
See http://scrapy.org/support/
See http
s
://scrapy.org/support/
debian/control
浏览文件 @
108f8c4f
...
@@ -4,7 +4,7 @@ Priority: optional
...
@@ -4,7 +4,7 @@ Priority: optional
Maintainer: Scrapinghub Team <info@scrapinghub.com>
Maintainer: Scrapinghub Team <info@scrapinghub.com>
Build-Depends: debhelper (>= 7.0.50), python (>=2.7), python-twisted, python-w3lib, python-lxml, python-six (>=1.5.2)
Build-Depends: debhelper (>= 7.0.50), python (>=2.7), python-twisted, python-w3lib, python-lxml, python-six (>=1.5.2)
Standards-Version: 3.8.4
Standards-Version: 3.8.4
Homepage: http://scrapy.org/
Homepage: http
s
://scrapy.org/
Package: scrapy
Package: scrapy
Architecture: all
Architecture: all
...
@@ -15,6 +15,6 @@ Conflicts: python-scrapy, scrapy-0.25
...
@@ -15,6 +15,6 @@ Conflicts: python-scrapy, scrapy-0.25
Provides: python-scrapy, scrapy-0.25
Provides: python-scrapy, scrapy-0.25
Description: Python web crawling and web scraping framework
Description: Python web crawling and web scraping framework
Scrapy is a fast high-level web crawling and web scraping framework,
Scrapy is a fast high-level web crawling and web scraping framework,
used to crawl websites and extract structured data from their pages.
used to crawl websites and extract structured data from their pages.
It can be used for a wide range of purposes, from data mining to
It can be used for a wide range of purposes, from data mining to
monitoring and automated testing.
monitoring and automated testing.
debian/copyright
浏览文件 @
108f8c4f
This package was debianized by the Scrapinghub team <info@scrapinghub.com>.
This package was debianized by the Scrapinghub team <info@scrapinghub.com>.
It was downloaded from http://scrapy.org
It was downloaded from http
s
://scrapy.org
Upstream Author: Scrapy Developers
Upstream Author: Scrapy Developers
...
@@ -14,10 +14,10 @@ All rights reserved.
...
@@ -14,10 +14,10 @@ All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice,
1. Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
documentation and/or other materials provided with the distribution.
...
...
docs/contributing.rst
浏览文件 @
108f8c4f
...
@@ -7,7 +7,7 @@ Contributing to Scrapy
...
@@ -7,7 +7,7 @@ Contributing to Scrapy
.. important::
.. important::
Double check you are reading the most recent version of this document at
Double check you are reading the most recent version of this document at
http://doc.scrapy.org/en/master/contributing.html
http
s
://doc.scrapy.org/en/master/contributing.html
There are many ways to contribute to Scrapy. Here are some of them:
There are many ways to contribute to Scrapy. Here are some of them:
...
...
docs/intro/overview.rst
浏览文件 @
108f8c4f
...
@@ -160,7 +160,7 @@ The next steps for you are to :ref:`install Scrapy <intro-install>`,
...
@@ -160,7 +160,7 @@ The next steps for you are to :ref:`install Scrapy <intro-install>`,
a full-blown Scrapy project and `join the community`_. Thanks for your
a full-blown Scrapy project and `join the community`_. Thanks for your
interest!
interest!
.. _join the community: http://scrapy.org/community/
.. _join the community: http
s
://scrapy.org/community/
.. _web scraping: https://en.wikipedia.org/wiki/Web_scraping
.. _web scraping: https://en.wikipedia.org/wiki/Web_scraping
.. _Amazon Associates Web Services: https://affiliate-program.amazon.com/gp/advertising/api/detail/main.html
.. _Amazon Associates Web Services: https://affiliate-program.amazon.com/gp/advertising/api/detail/main.html
.. _Amazon S3: https://aws.amazon.com/s3/
.. _Amazon S3: https://aws.amazon.com/s3/
...
...
docs/topics/practices.rst
浏览文件 @
108f8c4f
...
@@ -248,7 +248,7 @@ If you are still unable to prevent your bot getting banned, consider contacting
...
@@ -248,7 +248,7 @@ If you are still unable to prevent your bot getting banned, consider contacting
`commercial support`_.
`commercial support`_.
.. _Tor project: https://www.torproject.org/
.. _Tor project: https://www.torproject.org/
.. _commercial support: http://scrapy.org/support/
.. _commercial support: http
s
://scrapy.org/support/
.. _ProxyMesh: https://proxymesh.com/
.. _ProxyMesh: https://proxymesh.com/
.. _Google cache: http://www.googleguide.com/cached_pages.html
.. _Google cache: http://www.googleguide.com/cached_pages.html
.. _testspiders: https://github.com/scrapinghub/testspiders
.. _testspiders: https://github.com/scrapinghub/testspiders
...
...
docs/topics/selectors.rst
浏览文件 @
108f8c4f
...
@@ -86,7 +86,7 @@ To explain how to use the selectors we'll use the `Scrapy shell` (which
...
@@ -86,7 +86,7 @@ To explain how to use the selectors we'll use the `Scrapy shell` (which
provides interactive testing) and an example page located in the Scrapy
provides interactive testing) and an example page located in the Scrapy
documentation server:
documentation server:
http://doc.scrapy.org/en/latest/_static/selectors-sample1.html
http
s
://doc.scrapy.org/en/latest/_static/selectors-sample1.html
.. _topics-selectors-htmlcode:
.. _topics-selectors-htmlcode:
...
@@ -99,7 +99,7 @@ Here's its HTML code:
...
@@ -99,7 +99,7 @@ Here's its HTML code:
First, let's open the shell::
First, let's open the shell::
scrapy shell http://doc.scrapy.org/en/latest/_static/selectors-sample1.html
scrapy shell http
s
://doc.scrapy.org/en/latest/_static/selectors-sample1.html
Then, after the shell loads, you'll have the response available as ``response``
Then, after the shell loads, you'll have the response available as ``response``
shell variable, and its attached selector in ``response.selector`` attribute.
shell variable, and its attached selector in ``response.selector`` attribute.
...
...
docs/topics/shell.rst
浏览文件 @
108f8c4f
...
@@ -142,7 +142,7 @@ Example of shell session
...
@@ -142,7 +142,7 @@ Example of shell session
========================
========================
Here's an example of a typical shell session where we start by scraping the
Here's an example of a typical shell session where we start by scraping the
http://scrapy.org page, and then proceed to scrape the https://reddit.com
http
s
://scrapy.org page, and then proceed to scrape the https://reddit.com
page. Finally, we modify the (Reddit) request method to POST and re-fetch it
page. Finally, we modify the (Reddit) request method to POST and re-fetch it
getting an error. We end the session by typing Ctrl-D (in Unix systems) or
getting an error. We end the session by typing Ctrl-D (in Unix systems) or
Ctrl-Z in Windows.
Ctrl-Z in Windows.
...
@@ -154,7 +154,7 @@ shell works.
...
@@ -154,7 +154,7 @@ shell works.
First, we launch the shell::
First, we launch the shell::
scrapy shell 'http://scrapy.org' --nolog
scrapy shell 'http
s
://scrapy.org' --nolog
Then, the shell fetches the URL (using the Scrapy downloader) and prints the
Then, the shell fetches the URL (using the Scrapy downloader) and prints the
list of available objects and useful shortcuts (you'll notice that these lines
list of available objects and useful shortcuts (you'll notice that these lines
...
@@ -164,7 +164,7 @@ all start with the ``[s]`` prefix)::
...
@@ -164,7 +164,7 @@ all start with the ``[s]`` prefix)::
[s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s] crawler <scrapy.crawler.Crawler object at 0x7f07395dd690>
[s] crawler <scrapy.crawler.Crawler object at 0x7f07395dd690>
[s] item {}
[s] item {}
[s] request <GET http://scrapy.org>
[s] request <GET http
s
://scrapy.org>
[s] response <200 https://scrapy.org/>
[s] response <200 https://scrapy.org/>
[s] settings <scrapy.settings.Settings object at 0x7f07395dd710>
[s] settings <scrapy.settings.Settings object at 0x7f07395dd710>
[s] spider <DefaultSpider 'default' at 0x7f0735891690>
[s] spider <DefaultSpider 'default' at 0x7f0735891690>
...
@@ -182,7 +182,7 @@ After that, we can start playing with the objects::
...
@@ -182,7 +182,7 @@ After that, we can start playing with the objects::
>>> response.xpath('//title/text()').extract_first()
>>> response.xpath('//title/text()').extract_first()
'Scrapy | A Fast and Powerful Scraping and Web Crawling Framework'
'Scrapy | A Fast and Powerful Scraping and Web Crawling Framework'
>>> fetch("http://reddit.com")
>>> fetch("http
s
://reddit.com")
>>> response.xpath('//title/text()').extract()
>>> response.xpath('//title/text()').extract()
['reddit: the front page of the internet']
['reddit: the front page of the internet']
...
...
scrapy/_monkeypatches.py
浏览文件 @
108f8c4f
...
@@ -4,12 +4,12 @@ from six.moves import copyreg
...
@@ -4,12 +4,12 @@ from six.moves import copyreg
if
sys
.
version_info
[
0
]
==
2
:
if
sys
.
version_info
[
0
]
==
2
:
from
urlparse
import
urlparse
from
urlparse
import
urlparse
# workaround for http://bugs.python.org/issue7904 - Python < 2.7
# workaround for http
s
://bugs.python.org/issue7904 - Python < 2.7
if
urlparse
(
's3://bucket/key'
).
netloc
!=
'bucket'
:
if
urlparse
(
's3://bucket/key'
).
netloc
!=
'bucket'
:
from
urlparse
import
uses_netloc
from
urlparse
import
uses_netloc
uses_netloc
.
append
(
's3'
)
uses_netloc
.
append
(
's3'
)
# workaround for http://bugs.python.org/issue9374 - Python < 2.7.4
# workaround for http
s
://bugs.python.org/issue9374 - Python < 2.7.4
if
urlparse
(
's3://bucket/key?key=value'
).
query
!=
'key=value'
:
if
urlparse
(
's3://bucket/key?key=value'
).
query
!=
'key=value'
:
from
urlparse
import
uses_query
from
urlparse
import
uses_query
uses_query
.
append
(
's3'
)
uses_query
.
append
(
's3'
)
...
...
scrapy/core/downloader/contextfactory.py
浏览文件 @
108f8c4f
...
@@ -64,7 +64,7 @@ if twisted_version >= (14, 0, 0):
...
@@ -64,7 +64,7 @@ if twisted_version >= (14, 0, 0):
"""
"""
Twisted-recommended context factory for web clients.
Twisted-recommended context factory for web clients.
Quoting http://twistedmatrix.com/documents/current/api/twisted.web.client.Agent.html:
Quoting http
s
://twistedmatrix.com/documents/current/api/twisted.web.client.Agent.html:
"The default is to use a BrowserLikePolicyForHTTPS,
"The default is to use a BrowserLikePolicyForHTTPS,
so unless you have special requirements you can leave this as-is."
so unless you have special requirements you can leave this as-is."
...
@@ -100,6 +100,6 @@ else:
...
@@ -100,6 +100,6 @@ else:
def
getContext
(
self
,
hostname
=
None
,
port
=
None
):
def
getContext
(
self
,
hostname
=
None
,
port
=
None
):
ctx
=
ClientContextFactory
.
getContext
(
self
)
ctx
=
ClientContextFactory
.
getContext
(
self
)
# Enable all workarounds to SSL bugs as documented by
# Enable all workarounds to SSL bugs as documented by
# http
://www.openssl.org/docs/ssl
/SSL_CTX_set_options.html
# http
s://www.openssl.org/docs/manmaster/man3
/SSL_CTX_set_options.html
ctx
.
set_options
(
SSL
.
OP_ALL
)
ctx
.
set_options
(
SSL
.
OP_ALL
)
return
ctx
return
ctx
scrapy/crawler.py
浏览文件 @
108f8c4f
...
@@ -83,7 +83,7 @@ class Crawler(object):
...
@@ -83,7 +83,7 @@ class Crawler(object):
yield
defer
.
maybeDeferred
(
self
.
engine
.
start
)
yield
defer
.
maybeDeferred
(
self
.
engine
.
start
)
except
Exception
:
except
Exception
:
# In Python 2 reraising an exception after yield discards
# In Python 2 reraising an exception after yield discards
# the original traceback (see http://bugs.python.org/issue7563),
# the original traceback (see http
s
://bugs.python.org/issue7563),
# so sys.exc_info() workaround is used.
# so sys.exc_info() workaround is used.
# This workaround also works in Python 3, but it is not needed,
# This workaround also works in Python 3, but it is not needed,
# and it is slower, so in Python 3 we use native `raise`.
# and it is slower, so in Python 3 we use native `raise`.
...
...
scrapy/downloadermiddlewares/chunked.py
浏览文件 @
108f8c4f
...
@@ -11,7 +11,7 @@ warnings.warn("Module `scrapy.downloadermiddlewares.chunked` is deprecated, "
...
@@ -11,7 +11,7 @@ warnings.warn("Module `scrapy.downloadermiddlewares.chunked` is deprecated, "
class
ChunkedTransferMiddleware
(
object
):
class
ChunkedTransferMiddleware
(
object
):
"""This middleware adds support for chunked transfer encoding, as
"""This middleware adds support for chunked transfer encoding, as
documented in: http://en.wikipedia.org/wiki/Chunked_transfer_encoding
documented in: http
s
://en.wikipedia.org/wiki/Chunked_transfer_encoding
"""
"""
def
process_response
(
self
,
request
,
response
,
spider
):
def
process_response
(
self
,
request
,
response
,
spider
):
...
...
scrapy/downloadermiddlewares/httpcache.py
浏览文件 @
108f8c4f
...
@@ -75,7 +75,7 @@ class HttpCacheMiddleware(object):
...
@@ -75,7 +75,7 @@ class HttpCacheMiddleware(object):
return
response
return
response
# RFC2616 requires origin server to set Date header,
# RFC2616 requires origin server to set Date header,
# http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.18
# http
s
://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.18
if
'Date'
not
in
response
.
headers
:
if
'Date'
not
in
response
.
headers
:
response
.
headers
[
'Date'
]
=
formatdate
(
usegmt
=
1
)
response
.
headers
[
'Date'
]
=
formatdate
(
usegmt
=
1
)
...
...
scrapy/exporters.py
浏览文件 @
108f8c4f
...
@@ -188,7 +188,7 @@ class XmlItemExporter(BaseItemExporter):
...
@@ -188,7 +188,7 @@ class XmlItemExporter(BaseItemExporter):
self
.
xg
.
endElement
(
name
)
self
.
xg
.
endElement
(
name
)
self
.
_beautify_newline
()
self
.
_beautify_newline
()
# Workaround for http://bugs.python.org/issue17606
# Workaround for http
s
://bugs.python.org/issue17606
# Before Python 2.7.4 xml.sax.saxutils required bytes;
# Before Python 2.7.4 xml.sax.saxutils required bytes;
# since 2.7.4 it requires unicode. The bug is likely to be
# since 2.7.4 it requires unicode. The bug is likely to be
# fixed in 2.7.6, but 2.7.6 will still support unicode,
# fixed in 2.7.6, but 2.7.6 will still support unicode,
...
...
scrapy/extensions/httpcache.py
浏览文件 @
108f8c4f
...
@@ -70,8 +70,8 @@ class RFC2616Policy(object):
...
@@ -70,8 +70,8 @@ class RFC2616Policy(object):
return
True
return
True
def
should_cache_response
(
self
,
response
,
request
):
def
should_cache_response
(
self
,
response
,
request
):
# What is cacheable - http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec14.9.1
# What is cacheable - http
s
://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec14.9.1
# Response cacheability - http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.4
# Response cacheability - http
s
://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.4
# Status code 206 is not included because cache can not deal with partial contents
# Status code 206 is not included because cache can not deal with partial contents
cc
=
self
.
_parse_cachecontrol
(
response
)
cc
=
self
.
_parse_cachecontrol
(
response
)
# obey directive "Cache-Control: no-store"
# obey directive "Cache-Control: no-store"
...
@@ -163,7 +163,7 @@ class RFC2616Policy(object):
...
@@ -163,7 +163,7 @@ class RFC2616Policy(object):
def
_compute_freshness_lifetime
(
self
,
response
,
request
,
now
):
def
_compute_freshness_lifetime
(
self
,
response
,
request
,
now
):
# Reference nsHttpResponseHead::ComputeFreshnessLifetime
# Reference nsHttpResponseHead::ComputeFreshnessLifetime
# http
://dxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsHttpResponseHead.cpp#410
# http
s://dxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsHttpResponseHead.cpp#706
cc
=
self
.
_parse_cachecontrol
(
response
)
cc
=
self
.
_parse_cachecontrol
(
response
)
maxage
=
self
.
_get_max_age
(
cc
)
maxage
=
self
.
_get_max_age
(
cc
)
if
maxage
is
not
None
:
if
maxage
is
not
None
:
...
@@ -194,7 +194,7 @@ class RFC2616Policy(object):
...
@@ -194,7 +194,7 @@ class RFC2616Policy(object):
def
_compute_current_age
(
self
,
response
,
request
,
now
):
def
_compute_current_age
(
self
,
response
,
request
,
now
):
# Reference nsHttpResponseHead::ComputeCurrentAge
# Reference nsHttpResponseHead::ComputeCurrentAge
# http
://dxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsHttpResponseHead.cpp#366
# http
s://dxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsHttpResponseHead.cpp#658
currentage
=
0
currentage
=
0
# If Date header is not set we assume it is a fast connection, and
# If Date header is not set we assume it is a fast connection, and
# clock is in sync with the server
# clock is in sync with the server
...
@@ -414,7 +414,7 @@ class LeveldbCacheStorage(object):
...
@@ -414,7 +414,7 @@ class LeveldbCacheStorage(object):
def
parse_cachecontrol
(
header
):
def
parse_cachecontrol
(
header
):
"""Parse Cache-Control header
"""Parse Cache-Control header
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9
http
s
://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9
>>> parse_cachecontrol(b'public, max-age=3600') == {b'public': None,
>>> parse_cachecontrol(b'public, max-age=3600') == {b'public': None,
... b'max-age': b'3600'}
... b'max-age': b'3600'}
...
...
scrapy/extensions/telnet.py
浏览文件 @
108f8c4f
...
@@ -82,7 +82,7 @@ class TelnetConsole(protocol.ServerFactory):
...
@@ -82,7 +82,7 @@ class TelnetConsole(protocol.ServerFactory):
'prefs'
:
print_live_refs
,
'prefs'
:
print_live_refs
,
'hpy'
:
hpy
,
'hpy'
:
hpy
,
'help'
:
"This is Scrapy telnet console. For more info see: "
\
'help'
:
"This is Scrapy telnet console. For more info see: "
\
"http://doc.scrapy.org/en/latest/topics/telnetconsole.html"
,
"http
s
://doc.scrapy.org/en/latest/topics/telnetconsole.html"
,
}
}
self
.
crawler
.
signals
.
send_catch_log
(
update_telnet_vars
,
telnet_vars
=
telnet_vars
)
self
.
crawler
.
signals
.
send_catch_log
(
update_telnet_vars
,
telnet_vars
=
telnet_vars
)
return
telnet_vars
return
telnet_vars
scrapy/pipelines/files.py
浏览文件 @
108f8c4f
...
@@ -120,7 +120,7 @@ class S3FilesStore(object):
...
@@ -120,7 +120,7 @@ class S3FilesStore(object):
def
_get_boto_bucket
(
self
):
def
_get_boto_bucket
(
self
):
# disable ssl (is_secure=False) because of this python bug:
# disable ssl (is_secure=False) because of this python bug:
# http://bugs.python.org/issue5103
# http
s
://bugs.python.org/issue5103
c
=
self
.
S3Connection
(
self
.
AWS_ACCESS_KEY_ID
,
self
.
AWS_SECRET_ACCESS_KEY
,
is_secure
=
False
)
c
=
self
.
S3Connection
(
self
.
AWS_ACCESS_KEY_ID
,
self
.
AWS_SECRET_ACCESS_KEY
,
is_secure
=
False
)
return
c
.
get_bucket
(
self
.
bucket
,
validate
=
False
)
return
c
.
get_bucket
(
self
.
bucket
,
validate
=
False
)
...
@@ -268,7 +268,7 @@ class FilesPipeline(MediaPipeline):
...
@@ -268,7 +268,7 @@ class FilesPipeline(MediaPipeline):
def
__init__
(
self
,
store_uri
,
download_func
=
None
,
settings
=
None
):
def
__init__
(
self
,
store_uri
,
download_func
=
None
,
settings
=
None
):
if
not
store_uri
:
if
not
store_uri
:
raise
NotConfigured
raise
NotConfigured
if
isinstance
(
settings
,
dict
)
or
settings
is
None
:
if
isinstance
(
settings
,
dict
)
or
settings
is
None
:
settings
=
Settings
(
settings
)
settings
=
Settings
(
settings
)
...
...
scrapy/signalmanager.py
浏览文件 @
108f8c4f
...
@@ -55,7 +55,7 @@ class SignalManager(object):
...
@@ -55,7 +55,7 @@ class SignalManager(object):
The keyword arguments are passed to the signal handlers (connected
The keyword arguments are passed to the signal handlers (connected
through the :meth:`connect` method).
through the :meth:`connect` method).
.. _deferreds: http://twistedmatrix.com/documents/current/core/howto/defer.html
.. _deferreds: http
s
://twistedmatrix.com/documents/current/core/howto/defer.html
"""
"""
kwargs
.
setdefault
(
'sender'
,
self
.
sender
)
kwargs
.
setdefault
(
'sender'
,
self
.
sender
)
return
_signal
.
send_catch_log_deferred
(
signal
,
**
kwargs
)
return
_signal
.
send_catch_log_deferred
(
signal
,
**
kwargs
)
...
...
scrapy/templates/project/module/items.py.tmpl
浏览文件 @
108f8c4f
...
@@ -3,7 +3,7 @@
...
@@ -3,7 +3,7 @@
# Define here the models for your scraped items
# Define here the models for your scraped items
#
#
# See documentation in:
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html
# http
s
://doc.scrapy.org/en/latest/topics/items.html
import scrapy
import scrapy
...
...
scrapy/templates/project/module/middlewares.py.tmpl
浏览文件 @
108f8c4f
...
@@ -3,7 +3,7 @@
...
@@ -3,7 +3,7 @@
# Define here the models for your spider middleware
# Define here the models for your spider middleware
#
#
# See documentation in:
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/spider-middleware.html
# http
s
://doc.scrapy.org/en/latest/topics/spider-middleware.html
from scrapy import signals
from scrapy import signals
...
...
scrapy/templates/project/module/pipelines.py.tmpl
浏览文件 @
108f8c4f
...
@@ -3,7 +3,7 @@
...
@@ -3,7 +3,7 @@
# Define your item pipelines here
# Define your item pipelines here
#
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
# See: http
s
://doc.scrapy.org/en/latest/topics/item-pipeline.html
class ${ProjectName}Pipeline(object):
class ${ProjectName}Pipeline(object):
...
...
scrapy/templates/project/module/settings.py.tmpl
浏览文件 @
108f8c4f
...
@@ -5,9 +5,9 @@
...
@@ -5,9 +5,9 @@
# For simplicity, this file contains only settings considered important or
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
# commonly used. You can find more settings consulting the documentation:
#
#
# http://doc.scrapy.org/en/latest/topics/settings.html
# http
s
://doc.scrapy.org/en/latest/topics/settings.html
# http
://scrapy.readthedocs
.org/en/latest/topics/downloader-middleware.html
# http
s://doc.scrapy
.org/en/latest/topics/downloader-middleware.html
# http
://scrapy.readthedocs
.org/en/latest/topics/spider-middleware.html
# http
s://doc.scrapy
.org/en/latest/topics/spider-middleware.html
BOT_NAME = '$project_name'
BOT_NAME = '$project_name'
...
@@ -25,7 +25,7 @@ ROBOTSTXT_OBEY = True
...
@@ -25,7 +25,7 @@ ROBOTSTXT_OBEY = True
#CONCURRENT_REQUESTS = 32
#CONCURRENT_REQUESTS = 32
# Configure a delay for requests for the same website (default: 0)
# Configure a delay for requests for the same website (default: 0)
# See http
://scrapy.readthedocs
.org/en/latest/topics/settings.html#download-delay
# See http
s://doc.scrapy
.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
# The download delay setting will honor only one of:
...
@@ -45,31 +45,31 @@ ROBOTSTXT_OBEY = True
...
@@ -45,31 +45,31 @@ ROBOTSTXT_OBEY = True
#}
#}
# Enable or disable spider middlewares
# Enable or disable spider middlewares
# See http
://scrapy.readthedocs
.org/en/latest/topics/spider-middleware.html
# See http
s://doc.scrapy
.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#SPIDER_MIDDLEWARES = {
# '$project_name.middlewares.${ProjectName}SpiderMiddleware': 543,
# '$project_name.middlewares.${ProjectName}SpiderMiddleware': 543,
#}
#}
# Enable or disable downloader middlewares
# Enable or disable downloader middlewares
# See http
://scrapy.readthedocs
.org/en/latest/topics/downloader-middleware.html
# See http
s://doc.scrapy
.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#DOWNLOADER_MIDDLEWARES = {
# '$project_name.middlewares.${ProjectName}DownloaderMiddleware': 543,
# '$project_name.middlewares.${ProjectName}DownloaderMiddleware': 543,
#}
#}
# Enable or disable extensions
# Enable or disable extensions
# See http
://scrapy.readthedocs
.org/en/latest/topics/extensions.html
# See http
s://doc.scrapy
.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#EXTENSIONS = {
# 'scrapy.extensions.telnet.TelnetConsole': None,
# 'scrapy.extensions.telnet.TelnetConsole': None,
#}
#}
# Configure item pipelines
# Configure item pipelines
# See http
://scrapy.readthedocs
.org/en/latest/topics/item-pipeline.html
# See http
s://doc.scrapy
.org/en/latest/topics/item-pipeline.html
#ITEM_PIPELINES = {
#ITEM_PIPELINES = {
# '$project_name.pipelines.${ProjectName}Pipeline': 300,
# '$project_name.pipelines.${ProjectName}Pipeline': 300,
#}
#}
# Enable and configure the AutoThrottle extension (disabled by default)
# Enable and configure the AutoThrottle extension (disabled by default)
# See http://doc.scrapy.org/en/latest/topics/autothrottle.html
# See http
s
://doc.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
#AUTOTHROTTLE_START_DELAY = 5
...
@@ -82,7 +82,7 @@ ROBOTSTXT_OBEY = True
...
@@ -82,7 +82,7 @@ ROBOTSTXT_OBEY = True
#AUTOTHROTTLE_DEBUG = False
#AUTOTHROTTLE_DEBUG = False
# Enable and configure HTTP caching (disabled by default)
# Enable and configure HTTP caching (disabled by default)
# See http
://scrapy.readthedocs
.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
# See http
s://doc.scrapy
.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_DIR = 'httpcache'
...
...
scrapy/templates/project/scrapy.cfg
浏览文件 @
108f8c4f
# Automatically created by: scrapy startproject
# Automatically created by: scrapy startproject
#
#
# For more information about the [deploy] section see:
# For more information about the [deploy] section see:
# https://scrapyd.readthedocs.
org
/en/latest/deploy.html
# https://scrapyd.readthedocs.
io
/en/latest/deploy.html
[settings]
[settings]
default = ${project_name}.settings
default = ${project_name}.settings
...
...
scrapy/utils/defer.py
浏览文件 @
108f8c4f
...
@@ -57,7 +57,7 @@ def parallel(iterable, count, callable, *args, **named):
...
@@ -57,7 +57,7 @@ def parallel(iterable, count, callable, *args, **named):
"""Execute a callable over the objects in the given iterable, in parallel,
"""Execute a callable over the objects in the given iterable, in parallel,
using no more than ``count`` concurrent calls.
using no more than ``count`` concurrent calls.
Taken from: http://jcalderone.livejournal.com/24285.html
Taken from: http
s
://jcalderone.livejournal.com/24285.html
"""
"""
coop
=
task
.
Cooperator
()
coop
=
task
.
Cooperator
()
work
=
(
callable
(
elem
,
*
args
,
**
named
)
for
elem
in
iterable
)
work
=
(
callable
(
elem
,
*
args
,
**
named
)
for
elem
in
iterable
)
...
...
scrapy/utils/deprecate.py
浏览文件 @
108f8c4f
...
@@ -71,8 +71,8 @@ def create_deprecated_class(name, new_class, clsdict=None,
...
@@ -71,8 +71,8 @@ def create_deprecated_class(name, new_class, clsdict=None,
warnings
.
warn
(
msg
,
warn_category
,
stacklevel
=
2
)
warnings
.
warn
(
msg
,
warn_category
,
stacklevel
=
2
)
super
(
DeprecatedClass
,
cls
).
__init__
(
name
,
bases
,
clsdict_
)
super
(
DeprecatedClass
,
cls
).
__init__
(
name
,
bases
,
clsdict_
)
# see http://www.python.org/dev/peps/pep-3119/#overloading-isinstance-and-issubclass
# see http
s
://www.python.org/dev/peps/pep-3119/#overloading-isinstance-and-issubclass
# and http
://docs.python.org/2
/reference/datamodel.html#customizing-instance-and-subclass-checks
# and http
s://docs.python.org
/reference/datamodel.html#customizing-instance-and-subclass-checks
# for implementation details
# for implementation details
def
__instancecheck__
(
cls
,
inst
):
def
__instancecheck__
(
cls
,
inst
):
return
any
(
cls
.
__subclasscheck__
(
c
)
return
any
(
cls
.
__subclasscheck__
(
c
)
...
@@ -159,10 +159,10 @@ def update_classpath(path):
...
@@ -159,10 +159,10 @@ def update_classpath(path):
def
method_is_overridden
(
subclass
,
base_class
,
method_name
):
def
method_is_overridden
(
subclass
,
base_class
,
method_name
):
"""
"""
Return True if a method named ``method_name`` of a ``base_class``
Return True if a method named ``method_name`` of a ``base_class``
is overridden in a ``subclass``.
is overridden in a ``subclass``.
>>> class Base(object):
>>> class Base(object):
... def foo(self):
... def foo(self):
... pass
... pass
...
...
scrapy/utils/http.py
浏览文件 @
108f8c4f
...
@@ -11,7 +11,7 @@ def decode_chunked_transfer(chunked_body):
...
@@ -11,7 +11,7 @@ def decode_chunked_transfer(chunked_body):
decoded body.
decoded body.
For more info see:
For more info see:
http://en.wikipedia.org/wiki/Chunked_transfer_encoding
http
s
://en.wikipedia.org/wiki/Chunked_transfer_encoding
"""
"""
body
,
h
,
t
=
''
,
''
,
chunked_body
body
,
h
,
t
=
''
,
''
,
chunked_body
...
...
scrapy/utils/log.py
浏览文件 @
108f8c4f
...
@@ -154,7 +154,7 @@ class StreamLogger(object):
...
@@ -154,7 +154,7 @@ class StreamLogger(object):
"""Fake file-like stream object that redirects writes to a logger instance
"""Fake file-like stream object that redirects writes to a logger instance
Taken from:
Taken from:
http://www.electricmonk.nl/log/2011/08/14/redirect-stdout-and-stderr-to-a-logger-in-python/
http
s
://www.electricmonk.nl/log/2011/08/14/redirect-stdout-and-stderr-to-a-logger-in-python/
"""
"""
def
__init__
(
self
,
logger
,
log_level
=
logging
.
INFO
):
def
__init__
(
self
,
logger
,
log_level
=
logging
.
INFO
):
self
.
logger
=
logger
self
.
logger
=
logger
...
...
scrapy/utils/url.py
浏览文件 @
108f8c4f
...
@@ -47,7 +47,7 @@ def parse_url(url, encoding=None):
...
@@ -47,7 +47,7 @@ def parse_url(url, encoding=None):
def
escape_ajax
(
url
):
def
escape_ajax
(
url
):
"""
"""
Return the crawleable url according to:
Return the crawleable url according to:
http
://code.google.com/web/ajaxcrawling/docs/getting-started.html
http
s://developers.google.com/webmasters/ajax-crawling/docs/getting-started
>>> escape_ajax("www.example.com/ajax.html#!key=value")
>>> escape_ajax("www.example.com/ajax.html#!key=value")
'www.example.com/ajax.html?_escaped_fragment_=key%3Dvalue'
'www.example.com/ajax.html?_escaped_fragment_=key%3Dvalue'
...
...
sep/sep-001.rst
浏览文件 @
108f8c4f
...
@@ -61,7 +61,7 @@ ItemForm
...
@@ -61,7 +61,7 @@ ItemForm
--------
--------
Pros:
Pros:
- same API used for Items (see http://doc.scrapy.org/en/latest/topics/items.html)
- same API used for Items (see http
s
://doc.scrapy.org/en/latest/topics/items.html)
- some people consider setitem API more elegant than methods API
- some people consider setitem API more elegant than methods API
Cons:
Cons:
...
...
sep/sep-006.rst
浏览文件 @
108f8c4f
...
@@ -16,7 +16,7 @@ Motivation
...
@@ -16,7 +16,7 @@ Motivation
==========
==========
When you use Selectors in Scrapy, your final goal is to "extract" the data that
When you use Selectors in Scrapy, your final goal is to "extract" the data that
you've selected, as the [http://doc.scrapy.org/en/latest/topics/selectors.html
you've selected, as the [http
s
://doc.scrapy.org/en/latest/topics/selectors.html
XPath Selectors documentation] says (bolding by me):
XPath Selectors documentation] says (bolding by me):
When you’re scraping web pages, the most common task you need to perform is
When you’re scraping web pages, the most common task you need to perform is
...
@@ -58,7 +58,7 @@ As the name of the method for performing selection (the ``x`` method) is not
...
@@ -58,7 +58,7 @@ As the name of the method for performing selection (the ``x`` method) is not
descriptive nor mnemotechnic enough and clearly clashes with ``extract`` method
descriptive nor mnemotechnic enough and clearly clashes with ``extract`` method
(x sounds like a short for extract in english), we propose to rename it to
(x sounds like a short for extract in english), we propose to rename it to
`select`, `sel` (is shortness if required), or `xpath` after `lxml's
`select`, `sel` (is shortness if required), or `xpath` after `lxml's
<http://
codespeak.net/lxml
/xpathxslt.html>`_ ``xpath`` method.
<http://
lxml.de
/xpathxslt.html>`_ ``xpath`` method.
Bonus (ItemBuilder)
Bonus (ItemBuilder)
===================
===================
...
@@ -71,5 +71,5 @@ webpage or set of pages.
...
@@ -71,5 +71,5 @@ webpage or set of pages.
References
References
==========
==========
1. XPath Selectors (http://doc.scrapy.org/topics/selectors.html)
1. XPath Selectors (http
s
://doc.scrapy.org/topics/selectors.html)
2. XPath and XSLT with lxml (http://
codespeak.net/lxml
/xpathxslt.html)
2. XPath and XSLT with lxml (http://
lxml.de
/xpathxslt.html)
sep/sep-013.rst
浏览文件 @
108f8c4f
...
@@ -44,7 +44,7 @@ Overview of changes proposed
...
@@ -44,7 +44,7 @@ Overview of changes proposed
Most of the inconsistencies come from the fact that middlewares don't follow
Most of the inconsistencies come from the fact that middlewares don't follow
the typical
the typical
[http://twistedmatrix.com/projects/core/documentation/howto/defer.html
[http
s
://twistedmatrix.com/projects/core/documentation/howto/defer.html
deferred] callback/errback chaining logic. Twisted logic is fine and quite
deferred] callback/errback chaining logic. Twisted logic is fine and quite
intuitive, and also fits middlewares very well. Due to some bad design choices
intuitive, and also fits middlewares very well. Due to some bad design choices
the integration between middleware calls and deferred is far from optional. So
the integration between middleware calls and deferred is far from optional. So
...
...
sep/sep-017.rst
浏览文件 @
108f8c4f
...
@@ -13,7 +13,7 @@ SEP-017: Spider Contracts
...
@@ -13,7 +13,7 @@ SEP-017: Spider Contracts
The motivation for Spider Contracts is to build a lightweight mechanism for
The motivation for Spider Contracts is to build a lightweight mechanism for
testing your spiders, and be able to run the tests quickly without having to
testing your spiders, and be able to run the tests quickly without having to
wait for all the spider to run. It's partially based on the
wait for all the spider to run. It's partially based on the
[http://en.wikipedia.org/wiki/Design_by_contract Design by contract] approach
[http
s
://en.wikipedia.org/wiki/Design_by_contract Design by contract] approach
(hence its name) where you define certain conditions that spider callbacks must
(hence its name) where you define certain conditions that spider callbacks must
met, and you give example testing pages.
met, and you give example testing pages.
...
...
sep/sep-020.rst
浏览文件 @
108f8c4f
...
@@ -29,7 +29,7 @@ the rows and the further embedded ``<td>`` elements denoting the individual
...
@@ -29,7 +29,7 @@ the rows and the further embedded ``<td>`` elements denoting the individual
fields.
fields.
One pattern that is particularly well suited for auto-populating an Item Loader
One pattern that is particularly well suited for auto-populating an Item Loader
is the `definition list <http://www.w3.org/TR/html401/struct/lists.html#h-10.3>`_::
is the `definition list <http
s
://www.w3.org/TR/html401/struct/lists.html#h-10.3>`_::
<div class="geeks">
<div class="geeks">
<dl>
<dl>
...
...
setup.py
浏览文件 @
108f8c4f
...
@@ -29,7 +29,7 @@ if has_environment_marker_platform_impl_support():
...
@@ -29,7 +29,7 @@ if has_environment_marker_platform_impl_support():
setup
(
setup
(
name
=
'Scrapy'
,
name
=
'Scrapy'
,
version
=
version
,
version
=
version
,
url
=
'http://scrapy.org'
,
url
=
'http
s
://scrapy.org'
,
description
=
'A high-level Web Crawling and Web Scraping framework'
,
description
=
'A high-level Web Crawling and Web Scraping framework'
,
long_description
=
open
(
'README.rst'
).
read
(),
long_description
=
open
(
'README.rst'
).
read
(),
author
=
'Scrapy developers'
,
author
=
'Scrapy developers'
,
...
...
tests/__init__.py
浏览文件 @
108f8c4f
"""
"""
tests: this package contains all Scrapy unittests
tests: this package contains all Scrapy unittests
see http://doc.scrapy.org/en/latest/contributing.html#running-tests
see http
s
://doc.scrapy.org/en/latest/contributing.html#running-tests
"""
"""
import
os
import
os
...
...
tests/keys/example-com.conf
浏览文件 @
108f8c4f
# this is copied from http://stackoverflow.com/a/27931596
# this is copied from http
s
://stackoverflow.com/a/27931596
[
req
]
[
req
]
default_bits
=
2048
default_bits
=
2048
default_keyfile
=
server
-
key
.
pem
default_keyfile
=
server
-
key
.
pem
...
@@ -24,7 +24,7 @@ organizationName_default = Example, LLC
...
@@ -24,7 +24,7 @@ organizationName_default = Example, LLC
# Use a friendly name here because its presented to the user. The server's DNS
# Use a friendly name here because its presented to the user. The server's DNS
# names are placed in Subject Alternate Names. Plus, DNS names here is deprecated
# names are placed in Subject Alternate Names. Plus, DNS names here is deprecated
# by both IETF and CA/Browser Forums. If you place a DNS name here, then you
# by both IETF and CA/Browser Forums. If you place a DNS name here, then you
# must include the DNS name in the SAN too (otherwise, Chrome and others that
# must include the DNS name in the SAN too (otherwise, Chrome and others that
# strictly follow the CA/Browser Baseline Requirements will fail).
# strictly follow the CA/Browser Baseline Requirements will fail).
commonName
=
Common
Name
(
e
.
g
.
server
FQDN
or
YOUR
name
)
commonName
=
Common
Name
(
e
.
g
.
server
FQDN
or
YOUR
name
)
...
...
tox.ini
浏览文件 @
108f8c4f
# Tox (http
://tox.testrun.org
/) is a tool for running tests
# Tox (http
s://tox.readthedocs.io
/) is a tool for running tests
# in multiple virtualenvs. This configuration file will run the
# in multiple virtualenvs. This configuration file will run the
# test suite on all supported python versions. To use it, "pip install tox"
# test suite on all supported python versions. To use it, "pip install tox"
# and then run "tox" from this directory.
# and then run "tox" from this directory.
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录