Merge pull request #1767 from orangain/sitemap-robotstxt
[MRG+1] PY3: Fix SitemapSpider to extract sitemap urls from robots.txt properly
Showing
想要评论请 注册 或 登录
[MRG+1] PY3: Fix SitemapSpider to extract sitemap urls from robots.txt properly