Saturday 3 May 2014

multithreading and async-crawling

Due to the growing number of publications, crawl-time has increased dramatically. I spent the day experimenting with using multi-threading on the current implementation and using the Python Twisted library to crawl asynchronously. The latter results in a far greater speed-up, but would require a lot of code refactoring to implement.

1 comment:

  1. Top 10 best slots casinos for 2021 - SOL.EU
    Best Slots Casino: Best Real Money Slots Sites https://sol.edu.kg/ 2021 · herzamanindir Red Dog Casino: Best Overall Slots หารายได้เสริม Casino sporting100 For USA https://febcasino.com/review/merit-casino/ Players · Ignition Casino: Best Casino For Roulette

    ReplyDelete