I use Scrapy FakeUserAgent and keep getting this error on my Linux Server. I keep getting this error on the Linux server when I run multiple spiders concurrently. This error rarely happens on my own laptop. What should I do to avoid that? Do I have to raise the RAM or something? The server’s spec is 512MB RAM and 1
Tag: scrapy
Scrapy KeyError in Linux Server but not on Windows
My Scrapy works just fine on my local machine, Windows. Then I try to run it on my AWS Linux server, but I got this Why is that? How can I run it on my Linux Server? Answer Suddenly solved and I confused myself. I solve it by updating all the requirements using pip install -r requirements.txt. I added Scrapy
How to store screen output of Scrapy into a file in Linux
I often use to debug Scrapy project. However, when there are lot of items crawled, those log information will be embedded inside a lot of non-useful information. What I want is to store the screen output into a file, so that I can do search, do text editing, etc. I tried: but it doesn’t work Answer It seems that your
Installing scrapy (openSSL)
I’m using scrapy for couple of months now. I’ve been using it on the couple of machines and never had trouble with it. I used it on Windows xp and 7, I also used it on arch linux and never had trouble installing. Pretty much it was only pip install scrapy and done. Now I’m trying to set it up