How can I parse a website using Selenium … – Stack Overflow
New to programming and figured out how to navigate to where I need to go using Selenium. I’d like to parse the data now but not sure where to start. Can someone hold my hand a sec and point me in the right direction?
Any help appreciated –
asked Dec 19 ’12 at 20:06
Assuming you are on the page you want to parse, Selenium stores the source HTML in the driver’s page_source attribute. You would then load the page_source into BeautifulSoup as follows:
In : from bs4 import BeautifulSoup
In : from selenium import webdriver
In : driver = refox()
In : (”)
In : html = ge_source
In : soup = BeautifulSoup(html)
In : for tag in nd_all(‘title’):…. : print…. :…. :
answered Dec 19 ’12 at 20:19
RocketDonkeyRocketDonkey33. 7k7 gold badges75 silver badges83 bronze badges
As your question isn’t particularly concrete, here’s a simple example. To do something more useful read the BS docs. You will also find plenty of examples of selenium (and BS)usage here in SO.
from selenium import webdriver
from bs4 import BeautifulSoup
#do something useful
#prints all the links with corresponding text
for link in nd_all(‘a’):
print (‘href’, None), t_text()
answered Dec 19 ’12 at 20:18
Are you sure you want to use Selenium? For this reasons I used PyQt4, it’s very powerful, and you can do what ever you want.
I can give you a sample code, that I just wrote, just change url and you good to go:
#! /usr/bin/env python2. 7
from import *
from PyQt4. QtWebKit import *
import sys, signal
= (). currentFrame()
def _progress(self, progress):
print str(progress) + “%”
print “Load Finished”
html = unicode(())(‘utf-8’)
soup = BeautifulSoup(html)
if __name__ == “__main__”:
app = QApplication()
br = Browser()
url = QUrl(‘web site that can contain ‘)
if (, G_DFL):
answered Dec 19 ’12 at 20:14
VorVor28. 8k39 gold badges123 silver badges186 bronze badges
Not the answer you’re looking for? Browse other questions tagged python selenium beautifulsoup or ask your own question.
Web Scraping using Beautiful Soup and Selenium for dynamic …
In 10 minutes: Web Scraping with Beautiful Soup and …
Frequently Asked Questions about selenium bs4
Is Selenium better than Beautiful Soup?
Comparing selenium vs BeautifulSoup allows you to see that BeautifulSoup is more user-friendly and allows you to learn faster and begin web scraping smaller tasks easier. Selenium on the other hand is important when the target website has a lot of java elements in its code.Feb 10, 2021
What is Selenium Beautiful Soup?
When used together, Selenium and Beautiful Soup are powerful tools that allow the user to web scrape data efficiently and quickly.Mar 14, 2021
Can I use Beautiful Soup with Selenium?