Fixes #1593

[MRG+1] Disable SpiderState extension if no JOBDIR set

Use long classes names for enabled middlewares in startup logs

[MRG+1] Enable robots.txt handling by default for new projects.

For backwards compatibility reasons the default value is not changed.

[MRG+1] Disable CloseSpider extension if no CLOSPIDER_* setting set

[MRG+1] deprecated unused and untested code in scrapy.utils.datatypes

[MRG+1] Fix logging of enabled middlewares

Wrong middlewares list was being pretty-printed
(introduced in #1263)

[MRG+1] tests+doc for subdomains in offsite middleware

[MRG+1] PY3 exporters

[MRG+1] py3: test_pipeline_files and test_pipeline_images

[MRG+1] py3: reviewed passing test_spidermiddleware_httperror.py

[MRG+1] py3: fix test_mail

PY3 enable tests for scrapy parse command

scrapy parse command is already ported