How to use Python to Analyze Website Broken Links for SEO

What Broken Links Are and Why They Matter for SEO

Broken links, also called dead links, are URLs on a website that no longer lead to the intended destination. They show up for many reasons: a page was deleted, a URL structure changed, or an external site went offline. When a visitor clicks one, they usually land on a 404 error page instead of the content they expected.

These links are more than a small annoyance. Search engines like Google weigh the quality and functionality of links when they rank pages, so a site full of dead links can slip in the results. Ahrefs explains the concept well in its broken link glossary entry, and Google's own crawling and indexing guidance shows why healthy links keep your pages discoverable.

Here is why broken links work against you.

• User experience: Dead links frustrate visitors by sending them to pages that do not exist. That pushes bounce rates up, and search engines may read high bounce rates as a sign of weak content.
• Crawl efficiency: Search engine bots crawl your site to index it. Too many broken links disrupt that process and can leave parts of your site unindexed.
• Link equity loss: Backlinks from other sites pass authority to your pages. If a backlink points to a broken page, you lose the value it could have added.
• Ranking impact: Search engines favor sites that give a good experience. A site with many dead links looks less reliable and tends to rank lower.

• Deleted or moved pages40%
• External sites offline30%
• URL structure changes20%
• Typos in href values10%

If manual auditing feels slow, this is exactly the kind of repetitive work worth automating. Our guide to Python SEO automation covers more scripts like this one, and if you would rather hand the work off, our team of vetted SEO specialists can run these audits for you.

The Broken Link Analysis Workflow

Before touching code, it helps to see the full process end to end. The script you build follows these steps every time it runs.

1. 1
   Set up the environmentInstall Python plus the requests and beautifulsoup4 libraries.
2. 2
   Crawl a pageDownload the HTML of the page you want to check.
3. 3
   Extract linksParse the HTML and pull every href out of the anchor tags.
4. 4
   Resolve full URLsUse urljoin to turn relative paths into complete URLs.
5. 5
   Test each linkSend a request and read the HTTP status code.
6. 6
   Report and fixLog broken links, then correct or redirect them.

Each step maps to a small, readable block of Python. You do not need to be an expert to run it.

Setting Up the Python Environment

You need a working Python setup before the script will run. Three quick steps get you there.

1. Install Python. If you do not have it yet, download it from Python's official website and follow the installation instructions for your operating system.

2. Install the required libraries. Python's strength is its libraries. For link checking you need requests and beautifulsoup4. Install both with pip from your terminal or command prompt.

pip install requests
pip install beautifulsoup4

3. Choose an editor. You can write Python in any text editor, but an IDE like PyCharm, VS Code, or Jupyter Notebook makes life easier with syntax highlighting and debugging.

Writing the Python Script

With the environment ready, you can build the script piece by piece.

First, import the libraries you need.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

• requests makes the HTTP requests to web pages.
• BeautifulSoup parses the HTML and XML.
• urljoin builds full URLs from relative paths.

Next, define a small function that checks whether a single link works.

def check_link(url):
    try:
        response = requests.get(url, timeout=5)
        if response.status_code != 200:
            return False
        return True
    except requests.exceptions.RequestException:
        return False

This sends a request to each URL. It returns True when the page loads with status code 200 and False when it does not, or when the request fails entirely.

Now write a function that crawls a page, collects its links, and tests each one.

def find_broken_links(base_url):
    broken_links = []
    try:
        response = requests.get(base_url)
        soup = BeautifulSoup(response.text, 'html.parser')
        for link in soup.find_all('a'):
            url = link.get('href')
            if url:
                full_url = urljoin(base_url, url)
                if not check_link(full_url):
                    broken_links.append(full_url)
    except requests.exceptions.RequestException as e:
        print(f"Failed to access {base_url}: {e}")
    return broken_links

This requests the base URL, parses the HTML, finds every <a> tag, resolves each into a full URL, and checks it with check_link.

Finally, run the script.

if __name__ == "__main__":
    base_url = "https://example.com"  # Replace with your website URL
    broken_links = find_broken_links(base_url)
    if broken_links:
        print("Broken links found:")
        for link in broken_links:
            print(link)
    else:
        print("No broken links found.")

Replace https://example.com with the site you want to analyze. Save the file, open the folder where it lives, and run it.

python your_script_name.py

The script prints a list of broken links if it finds any. For a related automation, see how to check Google keyword ranking with Python.

Interpreting and Acting on the Results

A list of dead links is only useful if you do something with it. Here is what to do after each scan.

Result	Action	Why it matters
Broken internal link	Fix the URL or add a 301 redirect	Keeps users and bots on valid pages
Broken external link	Swap for a live source or remove it	Protects trust and relevance
Repeat offenders	Investigate the root cause	Stops the same links breaking again
Clean scan	Schedule the next check	Links break over time as content changes

1. Fix or redirect. For internal links, correct the URL or set up a 301 redirect. For external links, find an alternative resource or remove the link if it is no longer relevant.
2. Monitor regularly. Websites are dynamic, and links break as content is updated or removed. Rerun the script on a schedule to keep links healthy.
3. Improve user experience. A site free of dead ends feels smoother, which can lift engagement and conversions.
4. Bank the SEO benefits. Fixing broken links keeps your link structure intact, improves crawl efficiency, and makes sure link equity flows where it should. Backlinko's technical SEO research reinforces how much crawl health influences rankings.

Advanced Enhancements

The basic script works well, but a few upgrades make it far more capable, especially as your site grows.

MultithreadingHigh value for large sites, low effort to add

CSV and email reportsKeeps teams informed with little extra code

Search Console APICross-checks broken links against Google's index

Custom link filtersPrioritizes internal links and skips noise domains

1. Handling Large Websites

For small sites the basic script is fine. For thousands of pages, checking every link one at a time gets slow. Multithreading fixes that by testing many URLs at once. Python's concurrent.futures module makes it straightforward.

from concurrent.futures import ThreadPoolExecutor

def check_links_concurrently(base_url, max_workers=10):
    broken_links = []
    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        response = requests.get(base_url)
        soup = BeautifulSoup(response.text, 'html.parser')
        futures = []
        for link in soup.find_all('a'):
            url = link.get('href')
            if url:
                full_url = urljoin(base_url, url)
                futures.append(executor.submit(check_link, full_url))
        for future in futures:
            if not future.result():
                broken_links.append(full_url)
    return broken_links

You can also limit crawl depth and scope. Rather than every single page, focus on key pages like the homepage, category pages, and main blog articles.

2. Reporting and Notifications

Once you have found broken links, document them and get them in front of the right people. Two small additions handle this.

Export the results to a CSV file for records and analysis.

import csv

def export_to_csv(broken_links, filename="broken_links.csv"):
    with open(filename, mode='w', newline='') as file:
        writer = csv.writer(file)
        writer.writerow(["Broken Link"])
        for link in broken_links:
            writer.writerow([link])

Send an email alert when broken links turn up, using Python's smtplib.

import smtplib
from email.mime.text import MIMEText

def send_email_notification(broken_links, recipient_email):
    if not broken_links:
        return
    msg = MIMEText("\n".join(broken_links))
    msg['Subject'] = 'Broken Links Report'
    msg['From'] = 'your_email@example.com'
    msg['To'] = recipient_email
    with smtplib.SMTP('smtp.example.com') as server:
        server.login('your_email@example.com', 'your_password')
        server.send_message(msg)

3. Integrating with Other SEO Tools

Python pairs well with other tools. Connect it to the Google Search Console API and you can cross-reference the broken links your script finds with data from Google's index, catching discrepancies that could hurt rankings.

4. Customizing Link Checks

Not every broken link has the same priority. You may want to fix internal links before external ones, or skip domains like social media that do not matter for SEO. You can modify the script to focus only on internal links.

def find_internal_broken_links(base_url):
    broken_links = []
    try:
        response = requests.get(base_url)
        soup = BeautifulSoup(response.text, 'html.parser')
        for link in soup.find_all('a'):
            url = link.get('href')
            if url and base_url in url:
                full_url = urljoin(base_url, url)
                if not check_link(full_url):
                    broken_links.append(full_url)
    except requests.exceptions.RequestException as e:
        print(f"Failed to access {base_url}: {e}")
    return broken_links

Or skip certain domains entirely with an exclusion list.

def find_broken_links_with_exclusions(base_url, exclude_domains=None):
    if exclude_domains is None:
        exclude_domains = []
    broken_links = []
    try:
        response = requests.get(base_url)
        soup = BeautifulSoup(response.text, 'html.parser')
        for link in soup.find_all('a'):
            url = link.get('href')
            if url and all(domain not in url for domain in exclude_domains):
                full_url = urljoin(base_url, url)
                if not check_link(full_url):
                    broken_links.append(full_url)
    except requests.exceptions.RequestException as e:
        print(f"Failed to access {base_url}: {e}")
    return broken_links

5. Scheduling Regular Scans

Links break over time, so run the script on a schedule. Use a task scheduler such as cron on Linux or Task Scheduler on Windows.

0 2 * * 1 python /path/to/your/script.py

This example runs the script every Monday at 2 AM.

Manual checking

• Slow, page by page
• Easy to miss links
• Hard to repeat on schedule
• No record by default

vs

Python automation

• Scans many links per run
• Consistent and thorough
• Runs automatically with cron
• Exports CSV and email reports

Keeping Link Health Sustainable

Broken links can quietly drag down both your SEO and your visitors' experience. Regular checks and quick fixes keep your site healthy and high performing, and Python gives you a fast, repeatable way to do it. Start with the basic script, then layer on multithreading, reporting, and scheduling as your needs grow.

If your team does not have the bandwidth to run and maintain these scans, that is normal. Many businesses lean on outside help for ongoing technical upkeep. Seotal connects you with vetted SEO specialists, content managers, and website managers from Asia at 50 to 70 percent less than Western agencies. See how it works, review our transparent pricing, or get in touch to keep your links, and your rankings, in good shape.

Frequently asked questions

Which Python libraries do I need to check for broken links?

You need two: requests to send HTTP requests to each page, and beautifulsoup4 to parse the HTML and pull out the links. Install both with pip. The built-in urllib.parse module gives you urljoin, which turns relative paths into full URLs, and standard library modules like csv, smtplib, and concurrent.futures cover reporting and speed. No paid tools are required.

How does the script decide a link is broken?

The check_link function sends a request to a URL and reads the HTTP status code. A status of 200 means the page loaded, so the link is treated as working. Anything else, or a request that fails outright and raises an exception, is treated as broken and added to the results list. A five second timeout stops slow servers from hanging the whole scan.

How often should I run a broken link scan?

It depends on how often your content changes, but a weekly scan is a sensible default for most sites, which is why the cron example runs every Monday at 2 AM. Large or fast-moving sites may want daily checks, while small static sites can run monthly. The key is consistency, since links break gradually as pages move, get deleted, or external sites go offline.