All Questions
Tagged with python-2.7 scrapy
908
questions
1
vote
0
answers
43
views
Scrapy with Python 2.7 on wsl/Red Hat, errors all the way
So I need to run Scrapy at work. I cannot use python3 for the moment because the installation is messed up (long story).
I install scrapy using pip install scrapy.
The version with scrapy --version is ...
- 79
0
votes
3
answers
1k
views
ImportError: No module named importlib
I'm trying to install scrapy on python 2.7. The problem is that when it's time to compile it I get this error
$ sudo python2 -m pip install scrapy
DEPRECATION: Python 2.7 reached the end of its life ...
- 11
0
votes
1
answer
170
views
Run CrawlerProcess in Scrapy with Splash
I have a scrapy+splash file to crawl data. Now I want to run my scrapy file by script so I use CrawlerProcess. My file is like this:
import scrapy
from scrapy_splash import SplashRequest
from scrapy....
- 27
3
votes
2
answers
1k
views
Using Python2 and scrapy ImportError: cannot import name suppress
Hi am trying to run a scraper on ubuntu/windows machine .
I have installed scrapy version- Scrapy 1.8.0 on using python2.
I am able create a project, but when I run a scraper this error in shown.
...
- 31
0
votes
1
answer
86
views
Scrapy instance method mysteriously refusing to call another instance method
I'm using Scrapy to scrape a site that has a login page followed by a set of content pages with sequential integer IDs, pulled up as a URL parameter. This has been successfully running for a while, ...
- 123
0
votes
0
answers
206
views
ImportError: No module named robotparser
I'm trying to run a script but for some reasons I have this error showing up
scrapy crawl crawler -a url=https://www.montresmecaniques.com/ -o out.csv
Traceback (most recent call last):
File "/...
- 21
0
votes
0
answers
32
views
Broken github scrapy repo?
I've tried debugging this repo (https://github.com/dsuess/arXiv-vs-viXra) and have had some progress but I'm not sure what the issue is. I'm on Windows 10 OS and scrapy.py only runs on Python 2.7 to ...
- 11
0
votes
1
answer
374
views
Python - Web scraping using Scrapy
Just started learning web scraping using scrapy framework. I am trying to scrape reviews of a medicine from a medicinal website using the below code. But if i run "scrapy runspider spiders/...
- 3
0
votes
1
answer
196
views
Extract title from multiple urls in python scrapy
I am using multiple links but I not able to fetch title in each link:
import scrapy
from ..items import EbayurlItem
class StarterAutomotive(scrapy.Spider):
name = 'starterautomotive'
...
0
votes
3
answers
62
views
how i just extract the link for class = main-bullet using web craweling
i used this and using this i will extract all the in the web page but i want specific link which are in the main-bullet class.
import requests
import bs4
res = requests.get('http://www.hindisamay.com/...
- 17
0
votes
1
answer
323
views
Scrapy returns 400-error when trying to scrape Ajax call page
I am trying to scrape https://wegotthiscovered.com/reviews/ which uses Ajax pagination. I tried everything but it returns none or http-status code 400. Can anyone help in resolving this?
import json
...
0
votes
2
answers
225
views
How to pass custom parameters(such as -o) to scrapy crawler
I'm currently working on python2.7/Scrapy 1.8 project.
I work within a Docker container and using a
launchable.py:
import scrapy
from scrapy.crawler import CrawlerProcess
from spiders import ...
- 31
0
votes
1
answer
24
views
Check if all items from list have been sent for parsing
This is example code:
boxes = sel.xpath("//div[@class='lister-item mode-advanced']")
for box in boxes:
link = box.xpath(".//div[@class='lister-item-image float-left']/a/@href").extract_first()
...
- 41
0
votes
2
answers
1k
views
Docker build minimal image python 2.7 with scrapy
I'm doing a docker image with Python 2.7 Alpine and scrapy to export my script towards my team.
This is my dockerfile:
FROM python:2.7-alpine
WORKDIR /usr/src/app
RUN python -m pip install parse ...
- 31
0
votes
0
answers
104
views
Mac Catalina Python + Scrapy installation error
Robert@Macintosh ~ % cd ~/pythonfiles/fashionWebScraping/fashionWebScraping/spiders
Robert@Macintosh spiders % scrapy crawl -o rawdata_BOYNER.json -t jsonlines fashionBOYNER
2019-11-25 11:43:43 [...
- 383