diff --git a/docs/faq.rst b/docs/faq.rst index 3d2bd8d4d48ba9169f76aa3776b66527f3e119d4..b3412211ac76a30b9869d80e79d3600bd59db770 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -45,7 +45,7 @@ Did Scrapy "steal" X from Django? Probably, but we don't like that word. We think Django_ is a great open source project and an example to follow, so we've used it as an inspiration for -Scrapy. +Scrapy. We believe that, if something is already done well, there's no need to reinvent it. This concept, besides being one of the foundations for open source and free @@ -85,6 +85,8 @@ How can I simulate a user login in my spider? See :ref:`topics-request-response-ref-request-userlogin`. +.. _faq-bfo-dfo: + Does Scrapy crawl in breadth-first or depth-first order? -------------------------------------------------------- diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 052be4429bbcaa1bf8754d1a32dd38891f4dcde3..725345f2aab333a87e34067fe48a1f303d1f69b6 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -292,10 +292,14 @@ Scope: ``scrapy.spidermiddlewares.depth.DepthMiddleware`` An integer that is used to adjust the request priority based on its depth: -- **a positive value will decrease the priority** -- a negative value will increase priority - -If zero (default), no priority adjustment is made from depth. +- if zero (default), no priority adjustment is made from depth +- **a positive value will decrease the priority, i.e. higher depth + requests will be processed later** ; this is commonly used when doing + breadth-first crawls (BFO) +- a negative value will increase priority, i.e., higher depth requests + will be processed sooner (DFO) + +See also: :ref:`faq-bfo-dfo` about tuning Scrapy for BFO or DFO. .. note::