Uruchamianie Scrapy ze skryptu - zawiesza się

Question

Próbuję uruchomić skrypty ze skryptu, jak omówiono here. Sugerował użycie fragmentu this, ale kiedy to zrobię, zawiesza się w nieskończoność. Zostało to zapisane w wersji .10; czy nadal jest kompatybilny z aktualną stajnią?

Answer 1

from scrapy import signals, log 
from scrapy.xlib.pydispatch import dispatcher 
from scrapy.crawler import CrawlerProcess 
from scrapy.conf import settings 
from scrapy.http import Request 

def handleSpiderIdle(spider): 
    '''Handle spider idle event.''' # http://doc.scrapy.org/topics/signals.html#spider-idle 
    print '\nSpider idle: %s. Restarting it... ' % spider.name 
    for url in spider.start_urls: # reschedule start urls 
     spider.crawler.engine.crawl(Request(url, dont_filter=True), spider) 

mySettings = {'LOG_ENABLED': True, 'ITEM_PIPELINES': 'mybot.pipeline.validate.ValidateMyItem'} # global settings http://doc.scrapy.org/topics/settings.html 

settings.overrides.update(mySettings) 

crawlerProcess = CrawlerProcess(settings) 
crawlerProcess.install() 
crawlerProcess.configure() 

class MySpider(BaseSpider): 
    start_urls = ['http://site_to_scrape'] 
    def parse(self, response): 
     yield item 

spider = MySpider() # create a spider ourselves 
crawlerProcess.queue.append_spider(spider) # add it to spiders pool 

dispatcher.connect(handleSpiderIdle, signals.spider_idle) # use this if you need to handle idle event (restart spider?) 

log.start() # depends on LOG_ENABLED 
print "Starting crawler." 
crawlerProcess.start() 
print "Crawler stopped." 

UPDATE:

Jeśli trzeba mieć również ustawienia na pająka patrz poniższy przykład:

for spiderConfig in spiderConfigs: 
    spiderConfig = spiderConfig.copy() # a dictionary similar to the one with global settings above 
    spiderName = spiderConfig.pop('name') # name of the spider is in the configs - i can use the same spider in several instances - giving them different names 
    spiderModuleName = spiderConfig.pop('spiderClass') # module with the spider is in the settings 
    spiderModule = __import__(spiderModuleName, {}, {}, ['']) # import that module 
    SpiderClass = spiderModule.Spider # spider class is named 'Spider' 
    spider = SpiderClass(name = spiderName, **spiderConfig) # create the spider with given particular settings 
    crawlerProcess.queue.append_spider(spider) # add the spider to spider pool 

Przykład ustawienia w pliku, pająków:

name = punderhere_com  
allowed_domains = plunderhere.com 
spiderClass = scraper.spiders.plunderhere_com 
start_urls = http://www.plunderhere.com/categories.php?