How to export every URL in a sitemap to CSV, then get only the new ones

Step 1: find the sitemap

• robots.txt first. Open https://example.com/robots.txt and look for Sitemap: lines. A site can list several.
• Common locations if robots.txt names none: /sitemap.xml, /sitemap_index.xml (Yoast and Rank Math), /wp-sitemap.xml (WordPress core since 5.5), /sitemap.xml.gz.
• Expect an index. One sitemap file may hold at most 50,000 URLs and 50 MB uncompressed, so most real sites publish a <sitemapindex> that points to child files (posts, pages, products, images, news). Indexes can nest.

Quick one-liner (small, flat sitemaps)

curl -s https://example.com/sitemap.xml | grep -o '<loc>[^<]*</loc>' | sed 's/<[^>]*>//g' > urls.txt

This works when the file is a plain <urlset>. On an index it only returns the child sitemap files. On wordpress.org (2026-10-12) the first three lines were sitemap-1.xml, image-sitemap-1.xml and video-sitemap-1.xml, not pages. It also misses lastmod, gzip files and entities such as &.

A Python script that follows indexes and gzip

Standard library only. It recurses into sitemap indexes, unzips .gz files and writes url,lastmod,sitemap rows:

import csv, gzip, sys, urllib.request
import xml.etree.ElementTree as ET

NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}"

def fetch(url):
    req = urllib.request.Request(url, headers={"User-Agent": "sitemap-to-csv/1.0"})
    data = urllib.request.urlopen(req, timeout=30).read()
    return gzip.decompress(data) if data[:2] == b"\x1f\x8b" else data

def walk(url, rows, depth=0):
    root = ET.fromstring(fetch(url))
    if root.tag == NS + "sitemapindex" and depth < 5:
        for loc in root.iter(NS + "loc"):
            walk(loc.text.strip(), rows, depth + 1)
    else:
        for u in root.iter(NS + "url"):
            rows.append([u.findtext(NS + "loc", "").strip(), u.findtext(NS + "lastmod", "").strip(), url])

rows = []
walk(sys.argv[1], rows)
w = csv.writer(sys.stdout)
w.writerow(["url", "lastmod", "sitemap"])
w.writerows(rows)

python3 sitemap_to_csv.py https://wordpress.org/sitemap.xml > wordpress.csv

On 2026-10-12 that produced 812 URLs for wordpress.org. Pointed at apify.com it returned about 702,000 URLs, all held in memory. Where it falls short:

• It does not read robots.txt or guess locations, and it skips plain-text and RSS sitemaps.
• One malformed file or a single 429 stops the whole export; there are no retries.
• Image, video, news and hreflang data are ignored.
• It has no memory, so it cannot tell you what changed since yesterday.

Export without code

1. Open XML Sitemap Extractor to CSV & New-URL Monitor on Apify.
2. Enter domains (example.com), site URLs or direct sitemap URLs, one per line. Up to 10,000 sites per run.
3. Keep All URLs mode for a one-off export.
4. Optionally add include or exclude regex patterns (/products/, /blog/), a date filter (7 days or 2026-09-01) or the HTTP status check.
5. Click Start, then export the Output tab as CSV or Excel.

Discovery is automatic: Sitemap: lines from robots.txt first, then the common paths (only those robots.txt allows). It follows indexes up to 6 levels, reads gzip, text and RSS/Atom sitemaps, recovers URLs from broken XML and streams large files, so 50,000-URL sitemaps fit in normal memory. In our tests it read about 9,000 URLs from 8 inputs in 4 seconds.

A row from TechCrunch's news sitemap (real run, 2026-10-03):

{
  "site": "techcrunch.com",
  "url": "https://techcrunch.com/2026/10/03/federal-judge-calls-flock-indiscriminate-mass-surveillance/",
  "changeType": null,
  "sitemapUrl": "https://techcrunch.com/news-sitemap.xml",
  "news": {
    "publicationName": "TechCrunch",
    "language": "en",
    "publicationDate": "2026-10-03T19:33:15Z",
    "title": "Federal judge calls Flock ‘indiscriminate mass surveillance’"
  },
  "extractedAt": "2026-10-03T23:54:11+00:00"
}

Get only new, removed and updated URLs every day

Switch Mode to Changes only and schedule the actor (Apify Console, Schedules) with the same input. The first run saves each site's URL list in a named key-value store in your Apify account; every later run compares against it and outputs only the differences, each tagged with changeType:

{"url": "https://example.com/blog/c", "changeType": "new",            "lastmod": "2026-10-03", "previousLastmod": null}
{"url": "https://example.com/blog/a", "changeType": "removed",        "lastmod": null,         "previousLastmod": "2026-09-01"}
{"url": "https://example.com/blog/b", "changeType": "lastmodChanged", "lastmod": "2026-10-02", "previousLastmod": "2026-09-02"}

Recommended input for a competitor monitor:

{
  "sites": [
    "techcrunch.com",
    "allbirds.com",
    "https://wordpress.org/sitemap.xml"
  ],
  "mode": "changes",
  "emitBaselineOnFirstRun": false,
  "includePatterns": [
    "/products/",
    "/20\\d\\d/"
  ],
  "excludePatterns": [
    "/live/"
  ],
  "snapshotStoreName": "competitor-monitor"
}

Worked example (2026-10-03). A baseline over the Guardian, BBC, dir.bg, Allbirds and wordpress.org saved 42,985 URLs. The next run output exactly 1 row: a new Guardian article. A local test with a controlled sitemap returned exactly 1 new, 1 removed and 1 updated URL.

Details that keep the alerts clean:

• Fetch-time lastmod. Shopify product sitemaps and some news sitemaps stamp every URL with the time of the request. The actor detects this, sets lastmodIsFetchTime: true and does not report those URLs as updated.
• Partial failures. If some sitemap files fail to load, removed URLs are not reported for that run and the saved list is merged, not replaced, so you do not get false "removed" alerts.
• Silent first run. With emitBaselineOnFirstRun: false the first run only saves the list. Turn it on if you also want the full export once.
• Separate monitors. Use a different snapshotStoreName for each independent monitor.

What each row contains

A per-site SUMMARY record in the run's key-value store lists sitemaps found, parsed and failed, URLs listed and matched, change counts and any issues (for example NO_SITEMAP_FOUND).

Honest limits

• Only what the sitemap lists. It does not crawl links, so pages missing from the sitemap are missing from the export.
• lastmodChanged is only as good as the site's dates. Many CMSs update <lastmod> correctly; some never do.
• No bypassing. It identifies itself honestly, uses no proxy and records blocked_by_site when a site refuses it (403, persistent 429 or a challenge page). Some sites refuse any User-Agent that mentions Apify; the Guardian did in our 2026-10-04 test.
• Memory for huge sites. 100,000 URLs from bbc.com peaked at 165 MB. Plan roughly 1 GB of run memory per million URLs processed at once.
• Private and internal addresses (127.0.0.1, 10.0.0.0/8) are refused.

All input fields

Generated from the actor's live input schema (build 1.0.3).

Call it from code

Start a run and get the results in one HTTP request with the Apify API (your token is under Console, Settings, API & Integrations):

curl -X POST "https://api.apify.com/v2/acts/adamhaleworks~sitemap-extractor-change-monitor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"sites":["apify.com"],"maxUrlsPerSite":5000}'

The run's Output tab offers 2 ready-made views; every view exports to CSV, Excel, JSON or XML:

• URLs: site, url, changeType, lastmod, previousLastmod, changefreq, priority, statusCode, redirectLocation, imageCount, sitemapUrl
• News sitemap entries: site, url, news.title, news.publicationDate, news.publicationName, news.language, lastmod

What it costs

With the planned prices a monitor is cheap after the first run, because only changed URLs are output: the 5-site example above would have cost about $0.01 per daily run, and its baseline $0.01 with the silent first run (or $8.61 with every URL output). Sites with no readable sitemap, blocked sites and invalid inputs would not be charged.

FAQ