Python Web Scraping with BeautifulSoup

Installing the Tools

You need two packages: requests to download the page, and beautifulsoup4 (imported as bs4) to parse and search its HTML.

pip install requests beautifulsoup4

Fetching and Parsing a Page

First fetch the raw HTML with requests, then hand its text to BeautifulSoup, which turns it into a searchable tree of tags you can navigate like a document.

import requests
from bs4 import BeautifulSoup

response = requests.get("https://example.com")
soup = BeautifulSoup(response.text, "html.parser")

print(soup.title.text)

find() and find_all()

find() returns the first matching tag on the page, while find_all() returns a list of every match, which you can then loop over like any other list.

import requests
from bs4 import BeautifulSoup

response = requests.get("https://example.com/jobs")
soup = BeautifulSoup(response.text, "html.parser")

first_heading = soup.find("h2")
print(first_heading.text)

job_titles = soup.find_all("h2", class_="job-title")
for job in job_titles:
    print(job.text.strip())

Using select() with CSS Selectors

If you already know CSS selectors, select() lets you find elements the same way, which is often shorter than chaining find calls together.

links = soup.select("div.job-card a")
for link in links:
    print(link.text.strip(), link.get("href"))

Notice that .text (or .get_text()) pulls out the readable text inside a tag, while .get("href") reads an attribute such as a link URL — these two patterns cover most scraping needs.