scrapy終了時ログ表示情報詳細

爬虫類終了ログ収集情報は、ログ収集または爬虫類監視として使用できます.

2019-08-05 08:22:04 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 1781,         #     
'downloader/exception_type_count/twisted.internet.error.ConnectionRefusedError': 34,            #       
'downloader/exception_type_count/twisted.internet.error.TimeoutError': 1724,       #     
'downloader/exception_type_count/twisted.web._newclient.ResponseFailed': 7,        #         
'downloader/exception_type_count/twisted.web._newclient.ResponseNeverReceived': 16,    #        
'downloader/request_bytes': 11834762,     #       
'downloader/request_count': 30080,        #     
'downloader/request_method_count/GET': 30080,    # GET    
'downloader/response_bytes': 762108730,    #       
'downloader/response_count': 28299,       #     
'downloader/response_status_count/200': 28299,    #     200   
'finish_reason': 'finished',    #       
'finish_time': datetime.datetime(2019, 8, 5, 8, 22, 4, 856127),    #       
'log_count/ERROR': 1,    #     ERROR    
'log_count/INFO': 52,    #     INFO    
'log_count/WARNING': 7,    #     WARNING    
'memusage/max': 315863040,    #       
'memusage/startup': 58642432,    #       
'request_depth_max': 1,    #       
'response_received_count': 28299,    #       
'retry/count': 1773,    #      
'retry/max_reached': 8,    #        
'retry/reason_count/twisted.internet.error.ConnectionRefusedError': 34,    #          
'retry/reason_count/twisted.internet.error.TimeoutError': 1716,    #          
'retry/reason_count/twisted.web._newclient.ResponseFailed': 7,    #          
'retry/reason_count/twisted.web._newclient.ResponseNeverReceived': 16,    #          
'scheduler/dequeued': 30080,    #      
'scheduler/dequeued/memory': 30080,
'scheduler/enqueued': 30080,
'scheduler/enqueued/memory': 30080,
'start_time': datetime.datetime(2019, 8, 5, 7, 59, 16, 803793)}    #       
2019-08-05 08:22:04 [scrapy.core.engine] INFO: Spider closed (finished)     #