Your XML sitemap is a list of the pages you're telling search engines to crawl and index. Every URL in it should return 200 OK. When some of them return 404, 410, server errors or redirects, you're sending crawlers to dead ends, and it usually means something else on the site is broken too.
The good news is that a sitemap is the easiest thing on your site to check. It's already a clean list of URLs. This guide shows how to check every URL in your sitemap for 404 errors in a few minutes: a shell one-liner, a more robust Python script, Screaming Frog and Google Search Console. Then it covers how to keep checking automatically.
Why 404s in your sitemap matter
Google's documentation is direct about what happens to error pages: "Google doesn't index URLs that return a 4xx status code, and URLs that are already indexed and return a 4xx status code are removed from the index" (Google Search Central).
A sitemap full of 404s causes three problems:
- Wasted crawling. Crawlers spend requests on pages that don't exist instead of the ones you want discovered. A few stale entries won't matter on a small site, but it adds up on a large one.
- A less trustworthy sitemap. A sitemap is a hint, not a command. A sitemap that's mostly accurate is a better hint than one that's half dead.
- A symptom of a bigger problem. A 404 in your sitemap means a page was deleted or moved while something still lists it. Often your navigation, internal links or other sites still point at that URL too.
Redirects belong on the list as well. A sitemap should list the final, canonical URL of each page, not an address that 301s somewhere else.
Step 1: find your sitemap (and every sitemap inside it)
Before checking anything, make sure you have the right file.
- Check robots.txt. Open
https://yoursite.com/robots.txtand look for aSitemap:line. It's the most reliable pointer. - Try the usual locations:
/sitemap.xml,/sitemap_index.xml(common with WordPress SEO plugins), and/wp-sitemap.xml(WordPress's built-in sitemap). - Look for a sitemap index. If the file contains
<sitemapindex>and lists other.xmlfiles rather than pages, it's an index. You need to check every sitemap it points to, not just the index itself. The script in Method 2 handles this for you.
A single sitemap can hold at most 50,000 URLs or 50 MB uncompressed. Bigger sites split into several files under an index (Google's sitemap guidelines). So on a large site, "the sitemap" is often dozens of files.
Method 1: a one-line check with curl
Good for: a quick check of a single, small sitemap from any Mac or Linux terminal.
curl -s https://example.com/sitemap.xml \
| grep -oE '<loc>[^<]+' | sed 's/<loc>//' \
| while read -r url; do
code=$(curl -o /dev/null -s -L --max-time 15 -w '%{http_code}' "$url")
[ "$code" != "200" ] && echo "$code $url"
doneIt downloads the sitemap and pulls out every <loc> URL. Then it requests each one, following redirects, and prints only the URLs that don't end in a 200. No output means every URL is fine.
Limitations to keep in mind:
- It doesn't open nested sitemaps. Run it on a sitemap index and it checks the child sitemap files, not the pages inside them.
- It doesn't work on gzipped sitemaps (
.xml.gz) without an extra decompress step. - Because of
-L, a URL that redirects to a working page shows as200. That's fine for finding 404s, but it hides redirects you should clean up. Drop the-Lto see301s and302s as well. - It checks one URL at a time, so thousands of URLs take a while.
Method 2: a Python script that handles sitemap indexes
Good for: real sites with multiple sitemaps, and anyone who wants to see redirects too.
This script uses only Python's standard library. It opens sitemap indexes and checks URLs a few at a time. It tries a fast HEAD request first and falls back to GET for servers that reject HEAD. It prints every URL that isn't a clean 200, including ones that redirect.
#!/usr/bin/env python3
"""Check every URL in a sitemap (or sitemap index) and report non-200 responses.
Usage: python3 check_sitemap.py https://example.com/sitemap.xml
"""
import sys
import urllib.request
import urllib.error
import xml.etree.ElementTree as ET
from concurrent.futures import ThreadPoolExecutor
NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
UA = "Mozilla/5.0 (compatible; sitemap-checker/1.0)"
def fetch(url, method="GET"):
req = urllib.request.Request(url, method=method, headers={"User-Agent": UA})
return urllib.request.urlopen(req, timeout=15)
def sitemap_urls(sitemap_url):
root = ET.fromstring(fetch(sitemap_url).read())
if root.tag.endswith("sitemapindex"): # an index: recurse into each child sitemap
for loc in root.findall("sm:sitemap/sm:loc", NS):
yield from sitemap_urls(loc.text.strip())
else:
for loc in root.findall("sm:url/sm:loc", NS):
yield loc.text.strip()
def check(url):
try:
with fetch(url, "HEAD") as r:
final = r.geturl()
return url, r.status, final if final != url else ""
except urllib.error.HTTPError as e:
if e.code in (403, 405): # some servers reject HEAD; retry with GET
try:
with fetch(url) as r:
return url, r.status, ""
except urllib.error.HTTPError as e2:
return url, e2.code, ""
except Exception as e2:
return url, f"ERROR {type(e2).__name__}", ""
return url, e.code, ""
except Exception as e: # timeouts, DNS failures, refused connections
return url, f"ERROR {type(e).__name__}", ""
if __name__ == "__main__":
urls = list(dict.fromkeys(sitemap_urls(sys.argv[1])))
print(f"Checking {len(urls)} URLs...", file=sys.stderr)
with ThreadPoolExecutor(max_workers=5) as pool:
for url, status, final in pool.map(check, urls):
if status != 200 or final:
note = f" -> redirected to {final}" if final else ""
print(f"{status}\t{url}{note}")Run it and save the results:
python3 check_sitemap.py https://example.com/sitemap_index.xml > sitemap-problems.tsvExample output:
404 https://example.com/blog/old-announcement
200 https://example.com/pricing/ -> redirected to https://example.com/pricing
ERROR URLError https://example.com/legacy-landingThe second line is a URL that works but redirects, so the sitemap should list the final address instead. max_workers=5 checks five URLs at once. That's fine for your own server, but don't raise it much if your host is small. Results are tab-separated, so they paste straight into a spreadsheet.
Method 3: Screaming Frog in list mode
Good for: people who already use Screaming Frog and want the results next to everything else it reports.
Screaming Frog SEO Spider can crawl a sitemap directly, without spidering the whole site. Switch to Mode → List, choose Upload → Download XML Sitemap, and enter your sitemap URL. If it's a sitemap index, the same option fetches every sitemap inside it. Screaming Frog then requests every listed URL and reports its status code (their walkthrough). Filter the Response Codes tab for client errors (4xx), server errors (5xx) and redirects (3xx).
The free version is limited to 500 URLs, and auditing a sitemap in list mode works in it. Larger sitemaps need a paid licence.
Method 4: Google Search Console
Good for: seeing which sitemap problems Google has actually run into.
In Search Console, the Sitemaps report shows whether Google could read each submitted sitemap. The Page indexing report lists URLs Google couldn't index, with reasons such as Not found (404) and Soft 404 (Search Console Help). Use the filter at the top of that report to limit it to submitted pages, or to one specific sitemap.
The catch is timing. Search Console reflects Google's last crawl of each URL, which could be weeks old. Use it to confirm what Google has seen, and use Methods 1–3 to see what's true today.
Sitemap checks vs. crawling your site: you need both
A sitemap check and a site crawl answer different questions:
| Sitemap check | HTML crawl (following links) | |
|---|---|---|
| Starts from | The URLs listed in your sitemap | Your homepage, following every link |
| Finds | Listed pages that are dead or redirected | Links on your pages that point to dead URLs |
| Catches orphan pages (not linked from anywhere) | Yes, if they're in the sitemap | No |
| Catches dead links in your content and navigation | No | Yes |
| Checks outbound links to other sites | No | Yes, if configured to |
| Speed | Fast: one request per listed URL | Slower: has to download and parse every page |
A sitemap check confirms the pages you say exist really do. A crawl confirms the links on those pages work. A clean sitemap can still sit on a site full of broken internal links, and the other way around. For the crawl side, see How to Find and Fix Broken Links on Your Website. If outbound affiliate links are what you care about, this guide has a script that uses your sitemap to list every affiliate link on your site. On WordPress, a link-checker plugin covers the crawl side from inside the dashboard. Compare your options here.
How to fix what you find
Work through the results by status:
404or410: the page is gone.- If the content moved, add a
301redirect from the old URL to the new one, and make sure the sitemap lists the new URL. - If it's gone for good, remove it from the sitemap and leave it returning
404or410. Google treats both the same way, and a clean "gone" is better than redirecting everything to the homepage. - Either way, search your site for internal links to the old URL and update them.
- If the content moved, add a
301or302: the page redirects. Replace the sitemap entry with the final URL. Common causes are trailing-slash mismatches,httpvshttps, andwwwvs non-www.5xxerrors: the server failed. Re-check later. If specific URLs fail consistently, they likely have a bug (a broken template, a database query that times out) worth fixing.- Timeouts and DNS errors: if it's your own domain, something is seriously wrong with hosting or DNS. If the sitemap lists URLs on an old domain, regenerate it.
Then fix the source, not just the symptom. Most CMSs and SEO plugins generate sitemaps automatically, so stale entries usually mean one of three things. A static sitemap.xml file was uploaded once and never updated. A plugin is set to include post types or archives that no longer exist. Or a cache or CDN is serving an old copy. Regenerate the sitemap, purge the cache, and run the check again.
Keep your sitemap URLs checked automatically
A sitemap check is a snapshot. Pages get deleted, products sell out, slugs get renamed, and next month's sitemap problems don't exist yet. Re-run the check after every deploy, migration or bulk content change. If your site changes often, check daily.
You can schedule either script with cron and send the output to Slack. How to Monitor Broken Links Automatically walks through a full setup. Or you can hand it off: Brokenlink imports every URL from your sitemap.xml in a few clicks. It rechecks each one every 24 hours and alerts you by email or Slack with the exact URL and error code when one starts failing. Plans start at $9/month for 1,000 URLs, with a 14-day free trial and no card required (pricing).
How often is enough depends on how fast your site changes. How Often Should You Check for Broken Links? has a schedule by site type.
The short version
- Every URL in your sitemap should return
200. Anything else is a dead end for crawlers, and often a sign of broken internal links too. - Find every sitemap via
robots.txt, and don't forget the child sitemaps inside a sitemap index. - Quick check: the
curlone-liner. Real sites: the Python script, which handles indexes and shows redirects. Screaming Frog's list mode and Search Console's Page indexing report fill in the rest. - Fix 404s with a redirect or by removing the entry. Replace redirected entries with the final URL. Then fix whatever keeps generating stale entries.
- Sitemaps go stale, so keep checking them, on a schedule or with a monitor that imports your sitemap.