All Questions
Tagged with python-3.x beautifulsoup
5,161
questions
0
votes
0
answers
20
views
Scraping Amazon Shopping Cart in Real time via a chrome extension
I am trying to build a chrome extension that in real time scrapes the amazon website as the user is on it and gets us the subtotal in the user's cart before they proceed to checkout. We will use this ...
-2
votes
2
answers
42
views
Extract an Image from a Web Page
Every day, I need to manually extract the central image from two URLs. I decided to automate this process and, with the help of ChatGPT, I have the following code
# %%
from datetime import datetime, ...
- 15
-1
votes
0
answers
27
views
How to dynamically fetch the price div from e-commerce site in any browser?
res = requests.get(url, timeout=5)
content = BeautifulSoup(res.content, "html.parser")
price_div = content.find('div', attrs={"class": "CxhGGd"}).text.strip()
price = int(...
-4
votes
1
answer
54
views
When parsing HTML using Beautiful Soup, children tags not there sometimes
Using Beautiful Soup, I am working on a pretty standard program that parses through HTML and retrieves data based on their CSS elements. The issue I am having is that when I retrieve the CSS tag in ...
- 1
0
votes
0
answers
49
views
Are there any techniques in getting past Javascript checks with BeautifulSoup?
So I have the following script:
#!/usr/bin/env python3
import requests
from bs4 import BeautifulSoup
def parse_marketwatch_calendar(url):
#page=requests.get(url).text
#soup=BeautifulSoup(page,...
- 115
-1
votes
1
answer
17
views
How to extract value from attribute with Hyphen in them using BeautifulSoup
I have the below code:
from bs4 import BeautifulSoup as bs
soup = bs(requests.get(url).content, "html.parser")
for img in tqdm(soup.find_all("img"), "Extracting images"):...
- 69
-1
votes
1
answer
57
views
Amazon Product Links Scraping Using BeautifulSoup and Request
I am scraping this Amazon page:
https://www.amazon.com/s?k=mercedes&crid=37ZCGOIJ0A8V4&sprefix=me%2Caps%2C1587&ref=nb_sb_noss_2
Here's my code:
import requests
from bs4 import ...
1
vote
1
answer
31
views
Merging two tables using beautiful soup to extract links
I would like to scrape the first two tables from the following website:
https://fbref.com/en/comps/22/Major-League-Soccer-Stats
The tables I need are the first two, titled "Eastern Conference&...
- 51
0
votes
1
answer
42
views
Trouble scraping a website address from a webpage using the requests module
I'm trying to scrape the website address of Yauatcha Riyadh from a webpage using the requests module, but I end up getting None. I can fetch the title and the phone number from that page, but I failed ...
- 170
1
vote
1
answer
49
views
How to scrape the accurate fields from HTML tables of varying lengths?
I'm trying to create a script to scrape a few fields from tables based on their headers. The problem is that all the tables are not of the same length, along with their headers.
Here are the HTML ...
- 170
1
vote
2
answers
69
views
Web Scraping with Python without pagination site
I scrape data from the website using Selenium and BS4 and save it to json file. Since there is no pagination structure, I use web driver with selenium, but before adding selenium, while my old code ...
- 25
1
vote
0
answers
18
views
Web Scraping a Javascript Login Form
I am new to this web scraping but been searching everywhere and cannot find how to solve this problem that i'm having of performing a post Request to a form and get suceesfull login. from what i ...
- 21
1
vote
1
answer
16
views
Python BS4 various HTML tags
I'm trying to reach a particular value while web scraping in Python using BS4. The webpage is https://openaccess.thecvf.com/CVPR2021?day=all
I was able to get the titles by the tag ptitle and shows ...
- 23
1
vote
1
answer
27
views
Python Web Scraping: code output: soup not defined
My Yahoo stocks Webscraping program giving me errors.
Please find code below
def getdata(symbol):
headers= {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like ...
0
votes
1
answer
16
views
how to extract this value within soup body
here is my python code that does a BeautifulSoup on a API Call response content:
soup = BeautifulSoup(resp.content, 'lxml')
and if I output the soup body, it looks like this:
<html>
<body&...
- 745