From d999e3f7a704c5009999573d9514a5a51bd8be13 Mon Sep 17 00:00:00 2001 From: Paul Tremberth Date: Wed, 27 Jan 2016 12:57:03 +0100 Subject: [PATCH] More explicit description of DEPTH_PRIORITY --- docs/faq.rst | 4 +++- docs/topics/settings.rst | 12 ++++++++---- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/docs/faq.rst b/docs/faq.rst index 3d2bd8d4d..b3412211a 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -45,7 +45,7 @@ Did Scrapy "steal" X from Django? Probably, but we don't like that word. We think Django_ is a great open source project and an example to follow, so we've used it as an inspiration for -Scrapy. +Scrapy. We believe that, if something is already done well, there's no need to reinvent it. This concept, besides being one of the foundations for open source and free @@ -85,6 +85,8 @@ How can I simulate a user login in my spider? See :ref:`topics-request-response-ref-request-userlogin`. +.. _faq-bfo-dfo: + Does Scrapy crawl in breadth-first or depth-first order? -------------------------------------------------------- diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 052be4429..725345f2a 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -292,10 +292,14 @@ Scope: ``scrapy.spidermiddlewares.depth.DepthMiddleware`` An integer that is used to adjust the request priority based on its depth: -- **a positive value will decrease the priority** -- a negative value will increase priority - -If zero (default), no priority adjustment is made from depth. +- if zero (default), no priority adjustment is made from depth +- **a positive value will decrease the priority, i.e. higher depth + requests will be processed later** ; this is commonly used when doing + breadth-first crawls (BFO) +- a negative value will increase priority, i.e., higher depth requests + will be processed sooner (DFO) + +See also: :ref:`faq-bfo-dfo` about tuning Scrapy for BFO or DFO. .. note:: -- GitLab