Architecture

Similarly to Scrapy architecture

1. Request (base.BaseHttpRequest) from queue (base.BaseQueue) by engine (base.BaseEngine) passed to middlewares (base.BaseMiddleware) and then executed by downloader (base.BaseDownloader). Downloader can process request in concurrent way and return deferred result (utils.Planned).
2. Response (base.BaseHttpResponse) passed back to middlewares in reverse order and then to the crawler (base.BaseCrawler). Crawler can process response in concurrent way and return deferred result like downloader.
3. Extracted data passed to item pipeline.
4. Next requests if they exists will be putted to the queue.