Leaks docs

[MRG] process robots.txt once

[MRG] scrapy.utils.misc.load_object should print full traceback

[MRG] DOC document from_crawler method for item pipelines

Fix bug for ".local" host name

Fixed minor typo

Also create a list of the crawlers before iterating them because crawlers are removed from the set once stopped

Fix #874 issue

It's necessary to put new list member in squared brackets (i.e. create new list) to merge lists properly, otherwise we will get result list with character elements instead of string element.

Removed unused 'load=False' parameter from walk_modules()

GSoC API cleanup

SgmlLinkExtractor - fix for parsing <area> tag with Unicode present

Fix typo